hazelcast / hazelcast-kubernetes Goto Github PK
View Code? Open in Web Editor NEWKubernetes Discovery for Hazelcast
License: Apache License 2.0
hazelcast-kubernetes's People
Stargazers
Watchers
Forkers
kobalski corentin59 atomfrede amoahcp yurybubnov mrahbar fiksn chrba oscerd matf iteeconsulting emrahkocaman codepfleger barnyrelph emre-aydin st0ne-dot-at amiranda-ob asciijungle yohanesws ygxue lburgazzoli e-medicus isurulucky zfl926 xqianwang googlielmo marwestermann gamoshi-archives avalcepina pratyushamenaka mistic92 tstrilka renatoapcosta chaldeos leszko mmuruganandam topikachu mesutcelik tkgreg idanbauer newbare william-tran nmarasoiu ade90036 flyhard bardo1 eminn kwart sumhua mauna-ai zxshinxz ptzagk klubii henrikarro mustafaiman gasgithub metaver5o holmistr yeshm cut-to-the-cheese saindavs geertvanheusden adhesivee xuxinran rzs840707 azatkhakimov enozcan diegoep kennymacleod hepyu mouli-sanketh-maturi rafalleszko sumnerib purplewe jkremser timmhirsens doytsujin alparslanavci rajivbansal2981 janvanryswyckcegeka npomaroli zpqrtbnk arjen-iw gefgitacc1 sudarsan-sridharan alexrogalskiy payara fawazghali sadilchamishka xupengtony isabella232 kambaa linyanshi nagu002 ismart-yuxi anonymous-cowardhazelcast-kubernetes's Issues
HZ not joining cluster in kubernetes
Hello,
I have built a k8s pod that contains HZ 3.7.2 with hazelcast-kubernetes-discovery-0.9.3.jar and dnsjava-2.1.7.jar on the classpath and have configured hazelcast with
<discovery-strategies>
<discovery-strategy enabled="true" class="com.noctarius.hazelcast.kubernetes.HazelcastKubernetesDiscoveryStrategy">
<properties>
<!--k8s service name-->
<property name="service-dns">hazelcast</property>
</properties>
</discovery-strategy>
</discovery-strategies>
When starting up two pods in my deployment, I see in both logs:
> 2016-11-14T15:44:52.629474000Z ########################################
2016-11-14T15:44:52.629621000Z # RUN_JAVA=
2016-11-14T15:44:52.629762000Z # JAVA_OPTS=-Dhazelcast.rest.enabled=true
2016-11-14T15:44:52.629918000Z # starting now....
2016-11-14T15:44:52.630059000Z ########################################
2016-11-14T15:44:52.630204000Z Process id for hazelcast instance is written to location: /opt/hazelcast/hazelcast_instance.pid
2016-11-14T15:44:52.920870000Z Nov 14, 2016 3:44:52 PM com.hazelcast.config.XmlConfigLocator
2016-11-14T15:44:52.921057000Z INFO: Loading 'hazelcast.xml' from working directory.
2016-11-14T15:44:53.330555000Z Nov 14, 2016 3:44:53 PM com.hazelcast.instance.DefaultAddressPicker
2016-11-14T15:44:53.330736000Z INFO: [LOCAL] [hlx] [3.7.2] Prefer IPv4 stack is true.
2016-11-14T15:44:53.355237000Z Nov 14, 2016 3:44:53 PM com.hazelcast.instance.DefaultAddressPicker
2016-11-14T15:44:53.355397000Z INFO: [LOCAL] [hlx] [3.7.2] Picked [10.36.0.5]:5701, using socket ServerSocket[addr=/0:0:0:0:0:0:0:0,localport=5701], bind any local is true
2016-11-14T15:44:53.375015000Z Nov 14, 2016 3:44:53 PM com.hazelcast.system
2016-11-14T15:44:53.375212000Z INFO: [10.36.0.5]:5701 [hlx] [3.7.2] Hazelcast 3.7.2 (20161004 - 540b01c) starting at [10.36.0.5]:5701
2016-11-14T15:44:53.375965000Z Nov 14, 2016 3:44:53 PM com.hazelcast.system
2016-11-14T15:44:53.376126000Z INFO: [10.36.0.5]:5701 [hlx] [3.7.2] Copyright (c) 2008-2016, Hazelcast, Inc. All Rights Reserved.
2016-11-14T15:44:53.376977000Z Nov 14, 2016 3:44:53 PM com.hazelcast.system
2016-11-14T15:44:53.377147000Z INFO: [10.36.0.5]:5701 [hlx] [3.7.2] Configured Hazelcast Serialization version : 1
2016-11-14T15:44:53.562107000Z Nov 14, 2016 3:44:53 PM com.hazelcast.spi.impl.operationservice.impl.BackpressureRegulator
2016-11-14T15:44:53.562288000Z INFO: [10.36.0.5]:5701 [hlx] [3.7.2] Backpressure is disabled
2016-11-14T15:44:53.994721000Z Nov 14, 2016 3:44:53 PM com.hazelcast.spi.discovery.integration.DiscoveryService
2016-11-14T15:44:53.994916000Z INFO: [10.36.0.5]:5701 [hlx] [3.7.2] Kubernetes Discovery properties: { service-dns: hazelcast, service-name: null, service-label: null, service-label-value: true, namespace: default}
2016-11-14T15:44:53.996166000Z Nov 14, 2016 3:44:53 PM com.hazelcast.spi.discovery.integration.DiscoveryService
2016-11-14T15:44:53.996319000Z INFO: [10.36.0.5]:5701 [hlx] [3.7.2] Kubernetes Discovery activated resolver: DnsEndpointResolver
2016-11-14T15:44:53.997551000Z Nov 14, 2016 3:44:53 PM com.hazelcast.core.LifecycleService
2016-11-14T15:44:53.997700000Z INFO: [10.36.0.5]:5701 [hlx] [3.7.2] [10.36.0.5]:5701 is STARTING
2016-11-14T15:44:54.117326000Z Nov 14, 2016 3:44:54 PM com.hazelcast.spi.impl.operationexecutor.impl.OperationExecutorImpl
2016-11-14T15:44:54.117504000Z INFO: [10.36.0.5]:5701 [hlx] [3.7.2] Starting 2 partition threads
2016-11-14T15:44:54.120001000Z Nov 14, 2016 3:44:54 PM com.hazelcast.spi.impl.operationexecutor.impl.OperationExecutorImpl
2016-11-14T15:44:54.120160000Z INFO: [10.36.0.5]:5701 [hlx] [3.7.2] Starting 3 generic threads (1 dedicated for priority tasks)
2016-11-14T15:44:54.127213000Z Nov 14, 2016 3:44:54 PM com.hazelcast.nio.tcp.nonblocking.NonBlockingIOThreadingModel
2016-11-14T15:44:54.127368000Z INFO: [10.36.0.5]:5701 [hlx] [3.7.2] TcpIpConnectionManager configured with Non Blocking IO-threading model: 3 input threads and 3 output threads
2016-11-14T15:44:54.141210000Z Nov 14, 2016 3:44:54 PM com.hazelcast.instance.Node
2016-11-14T15:44:54.141367000Z WARNING: [10.36.0.5]:5701 [hlx] [3.7.2] No join method is enabled! Starting standalone.
2016-11-14T15:44:54.168042000Z Nov 14, 2016 3:44:54 PM com.hazelcast.core.LifecycleService
2016-11-14T15:44:54.168217000Z INFO: [10.36.0.5]:5701 [hlx] [3.7.2] [10.36.0.5]:5701 is STARTED
So far, so good. One pod is on 10.36.0.5, the other on 10.32.0.6. If I shell into a pod in the cluster and do an nslookup on the service name 'hazelcast', I get both addresses:
> nslookup hazelcast
Name: hazelcast
Address 1: 10.36.0.5
Address 2: 10.32.0.6
But when I query the cluster status from either of the hazelcast pods, I see that they are not joined:
curl 10.32.0.6:5701/hazelcast/rest/cluster
Members [1] {
Member [10.32.0.6]:5701 - f0afb098-55d7-428c-bc95-768760029590 this
}
curl 10.36.0.5:5701/hazelcast/rest/cluster
Members [1] {
Member [10.36.0.5]:5701 - dbba32b7-353d-400e-820c-1ce95d29079d this
}
So the HZ instances are starting with dns lookup enabled, but are not joining. What am I doing wrong?
regards
Peter
How to make non-ready pods discoverable by the plugin?
If two pods are initialized in parallel they don't see each other (because the Readiness Probe returns false) and form two separate clusters. Is there any way to prevent this behavior?
issues with kubernetes and preparing blog post with a sample
see the discussion here: https://gitter.im/hazelcast/hazelcast?at=5ae018a8109bb04332fb0f0d
https://blog.hazelcast.com/deploy-monitor/ could be outdated so we need to update it or just need to write a new one along with a code-samples...
Openshift example can be reused here.
https://github.com/hazelcast/hazelcast-code-samples/tree/master/hazelcast-integration/openshift
How to connect a client to a cluster
I have setup my cluster using the service-name option which works fine and multiple hazelcast nodes discover each other. However I can't figure out how to get my clients (spring boot java apps) to connect to all members in the cluster other than by specifying ip address which obviously does not fit with the K8 model. Can somebody advice/add to the docs how to connect a client that is running in the same K8 cluster?
ClassNotFoundException: io.fabric8.kubernetes.client.EditableConfig
Hi trying to deploy a cluster of Hazelcast in k8s using this library.
My understanding is that I need to build a new Docker image for Hazelcast that contains this jar file and a new config and deploy it in k8s.
So I did the following:
change the Dockerfile
FROM openjdk:8-jre
ENV HZ_VERSION 3.9
...
ADD http://central.maven.org/maven2/com/hazelcast/hazelcast-kubernetes/1.0.0/hazelcast-kubernetes-1.0.0.jar $HZ_HOME
ADD http://central.maven.org/maven2/io/fabric8/kubernetes-client/3.1.0/kubernetes-client-3.1.0.jar $HZ_HOME
ADD hazelcast.xml $HZ_HOMEadd the jar to the classpath in server.sh
export CLASSPATH=$HAZELCAST_HOME/hazelcast-kubernetes-1.0.0.jar:$CLASSPATH
export CLASSPATH=$HAZELCAST_HOME/kubernetes-client-3.1.0.jar:$CLASSPATH
and use the config hazelcast.xml
<hazelcast xsi:schemaLocation="http://www.hazelcast.com/schema/config hazelcast-config-2.0.xsd"
xmlns="http://www.hazelcast.com/schema/config"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<properties>
</properties>
<network>
<join>
<!-- deactivate normal discovery -->
<multicast enabled="false"/>
<tcp-ip enabled="false"/>
<!-- activate the Kubernetes plugin -->
<discovery-strategies>
<discovery-strategy enabled="true"
class="com.hazelcast.kubernetes.HazelcastKubernetesDiscoveryStrategy">
<properties>
<!-- configure discovery service API lookup -->
<property name="service-name">${hazelcast.k8s.service}</property>
<property name="namespace">${hazelcast.k8s.namespace}</property>
</properties>
</discovery-strategy>
</discovery-strategies>
</join>
</network>
</hazelcast>When the pod start I get the logging message:
Kubernetes Discovery properties: { service-dns: null, service-dns-timeout: 5, service-name: hazelcast, service-label: null, service-label-value: true, namespace: test, kubernetes-master: https://kubernetes.default.svc}
but then just after that I get:
Exception in thread "main" java.lang.NoClassDefFoundError: io/fabric8/kubernetes/client/EditableConfig
at com.hazelcast.kubernetes.HazelcastKubernetesDiscoveryStrategy.<init>(HazelcastKubernetesDiscoveryStrategy.java:75)
at com.hazelcast.kubernetes.HazelcastKubernetesDiscoveryStrategyFactory.newDiscoveryStrategy(HazelcastKubernetesDiscoveryStrategyFactory.java:56)
What I cannot understand is that EditableConfig is an old deprecated class from kubernetes-client. I'm using the latest version of the client so I don't understand how the old client would get into the classpath
Am I doing the correct thing here? If so any idea how this exception could happen?
SocketException[Connection refused to address /172.16.60.55:5701]
Hello, I am having issues while trying to have hazelcast connecting to other nodes. Let me know if I am missing anything. I would assume that if should find other nodes and auto connect to it.
Here is my k8 deployment:
NAME: hazelcast
REVISION: 1
RELEASED: Wed Feb 28 13:46:56 2018
CHART: hazelcast-testhz-0.1.0
USER-SUPPLIED VALUES:
env:
gitCommit: f34wew2a5aer33ffas7a
springProfile: qa
image:
pullPolicy: Always
repository: hazelcast/hazelcast-kubernetes
tag: 3.10-BETA-1
ingress:
enabled: true
hosts:
- hazelcast.us-west-2.nonprod.cnqr.delivery
tls: - hosts:
- hazelcast.us-west-2.nonprod.cnqr.delivery
nginx:
configmap:
enableunderscore: true
pod:
initialDelaySeconds: 120
timeoutSeconds: 60
replicaCount: 3
resources:
limits:
cpu: 2
memory: 4Gi
requests:
cpu: 600m
memory: 256Mi
service:
annotations:
service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
service.beta.kubernetes.io/aws-load-balancer-internal: 0.0.0.0/0
externalPort: 5071
internalPort: 5071
name: hazelcast
namespace: testhz
type: ClusterIP
- hazelcast.us-west-2.nonprod.cnqr.delivery
COMPUTED VALUES:
env:
gitCommit: f34wew2a5aer33ffas7a
springProfile: qa
image:
pullPolicy: Always
repository: hazelcast/hazelcast-kubernetes
tag: 3.10-BETA-1
ingress:
enabled: true
hosts:
- hazelcast.us-west-2.nonprod.cnqr.delivery
tls: - hosts:
- hazelcast.us-west-2.nonprod.cnqr.delivery
nginx:
configmap:
enableunderscore: true
pod:
initialDelaySeconds: 120
timeoutSeconds: 60
replicaCount: 3
resources:
limits:
cpu: 2
memory: 4Gi
requests:
cpu: 600m
memory: 256Mi
service:
annotations:
service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
service.beta.kubernetes.io/aws-load-balancer-internal: 0.0.0.0/0
externalPort: 5071
internalPort: 5071
name: hazelcast
namespace: testhz
type: ClusterIP
- hazelcast.us-west-2.nonprod.cnqr.delivery
HOOKS:
MANIFEST:
Source: hazelcast-testhz/templates/nginx-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: hz-config
namespace: testhz
data:
HAZELCAST_KUBERNETES_NAMESPACE: testhz
HAZELCAST_KUBERNETES_SERVICE_NAME: hazelcast
Source: hazelcast-testhz/templates/service.yaml
apiVersion: v1
kind: Service
metadata:
name: hazelcast
labels:
app: hazelcast
spec:
type: NodePort
selector:
app: hazelcast
ports:
- protocol: TCP
port: 5701
name: hzport
Source: hazelcast-testhz/templates/deployment.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: hazelcast
namespace: testhz
labels:
app: hazelcast
chart: hazelcast-0.1.0
release: hazelcast-hazelcast
heritage: Tiller-hazelcast
spec:
replicas: 3
template:
metadata:
labels:
app: hazelcast
release: hazelcast-hazelcast
spec:
containers:
- name: hazelcast
image: "hazelcast/hazelcast-kubernetes:3.10-BETA-1"
imagePullPolicy: Always
ports:
- containerPort: 5071
livenessProbe:
exec:
command:
- "./liveness.sh"
initialDelaySeconds: 120
timeoutSeconds: 60
periodSeconds: 10
resources:
limits:
cpu: 2
memory: 4Gi
requests:
cpu: 600m
memory: 256Mi
Source: hazelcast-testhz/templates/ingress.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: hazelcast
namespace: testhz
labels:
app: hazelcast
chart: hazelcast-0.1.0
release: hazelcast-hazelcast
heritage: Tiller
annotations:
spec:
rules:
- host: hazelcast.us-west-2.nonprod.cnqr.delivery
http:
paths:
- path: /
backend:
serviceName: hazelcast
servicePort: 5071
tls:
- hosts:
- hazelcast.us-west-2.nonprod.cnqr.delivery
LAST DEPLOYED: Wed Feb 28 13:46:56 2018
NAMESPACE: testhz
STATUS: DEPLOYED
RESOURCES:
==> v1/Service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
hazelcast NodePort 192.168.143.243 5701:31195/TCP 1s
==> v1beta1/Deployment
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
hazelcast 3 3 3 0 1s
Kube logs
╰─ kcwizlog po/hazelcast-2366664114-202vc
Kubernetes Namespace: testhz
Kubernetes Service DNS:
########################################
RUN_JAVA=
JAVA_OPTS=
CLASSPATH=/:/opt/hazelcast/:/opt/hazelcast/external/*:
########################################
Checking custom configuration
no custom configuration found
Feb 28, 2018 9:47:15 PM com.hazelcast.config.XmlConfigLocator
INFO: Loading 'hazelcast.xml' from working directory.
Feb 28, 2018 9:47:15 PM com.hazelcast.config.AbstractXmlConfigHelper
WARNING: Name of the hazelcast schema location is incorrect, using default
Feb 28, 2018 9:47:16 PM com.hazelcast.instance.AddressPicker
INFO: [LOCAL] [dev] [3.10-BETA-1] Prefer IPv4 stack is true.
Feb 28, 2018 9:47:16 PM com.hazelcast.instance.AddressPicker
INFO: [LOCAL] [dev] [3.10-BETA-1] Picked [172.16.18.169]:5701, using socket ServerSocket[addr=/0.0.0.0,localport=5701], bind any local is true
Feb 28, 2018 9:47:16 PM com.hazelcast.system
INFO: [172.16.18.169]:5701 [dev] [3.10-BETA-1] Hazelcast 3.10-BETA-1 (20180221 - c9a445b) starting at [172.16.18.169]:5701
Feb 28, 2018 9:47:16 PM com.hazelcast.system
INFO: [172.16.18.169]:5701 [dev] [3.10-BETA-1] Copyright (c) 2008-2018, Hazelcast, Inc. All Rights Reserved.
Feb 28, 2018 9:47:16 PM com.hazelcast.system
INFO: [172.16.18.169]:5701 [dev] [3.10-BETA-1] Configured Hazelcast Serialization version: 1
Feb 28, 2018 9:47:16 PM com.hazelcast.spi.impl.operationservice.impl.BackpressureRegulator
INFO: [172.16.18.169]:5701 [dev] [3.10-BETA-1] Backpressure is disabled
Feb 28, 2018 9:47:16 PM com.hazelcast.spi.discovery.integration.DiscoveryService
INFO: [172.16.18.169]:5701 [dev] [3.10-BETA-1] Kubernetes Discovery properties: { service-dns: null, service-dns-timeout: 5, service-name: null, service-label: null, service-label-value: true, namespace: default, resolve-not-ready-addresses: false, kubernetes-master: https://kubernetes.default.svc}
Feb 28, 2018 9:47:16 PM com.hazelcast.spi.discovery.integration.DiscoveryService
INFO: [172.16.18.169]:5701 [dev] [3.10-BETA-1] Kubernetes Discovery: Bearer Token { eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJ3aXphcmRzIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImRlZmF1bHQtdG9rZW4tbWx6NGoiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGVmYXVsdCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6Ijk0MDNlNGJiLWMwZWEtMTFlNy05NTk4LTBhMGIxNzMwNDgwMiIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDp3aXphcmRzOmRlZmF1bHQifQ.WBNhlfnXJ6_1d3KfVitj6L_tHXizIn-OgOpZV6DbZjataKWY37VlJBToSdJTvODlOh6V9LvI7C7bQl66DEpuhr3netv7Z0AFPFk1nlOvgblVwRDUqDNI5tTayD3sr5oIVRNNOqljPPa89UIYwCkL8BKxmZwMIE97r_tFRWEW01psU8U68UgE-2nudMHBkyRtRlKlRJsMkU79N9Y2Fbd1DqsNrTtenVyO9ZeTwLMw7cuhwvHl6N4wDjerJDM8VV4yjv5YUkrFD9ID0GINt6lXHCeTxcESkKxfvYVUxCAEKs2ixyMXIskwhI_fNvzetVzMfH95Fnu153amuwrvVtpqIA }
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Feb 28, 2018 9:47:17 PM com.hazelcast.spi.discovery.integration.DiscoveryService
INFO: [172.16.18.169]:5701 [dev] [3.10-BETA-1] Kubernetes Discovery activated resolver: ServiceEndpointResolver
Feb 28, 2018 9:47:17 PM com.hazelcast.instance.Node
INFO: [172.16.18.169]:5701 [dev] [3.10-BETA-1] Activating Discovery SPI Joiner
Feb 28, 2018 9:47:17 PM com.hazelcast.spi.impl.operationexecutor.impl.OperationExecutorImpl
INFO: [172.16.18.169]:5701 [dev] [3.10-BETA-1] Starting 16 partition threads and 9 generic threads (1 dedicated for priority tasks)
Feb 28, 2018 9:47:17 PM com.hazelcast.internal.diagnostics.Diagnostics
INFO: [172.16.18.169]:5701 [dev] [3.10-BETA-1] Diagnostics disabled. To enable add -Dhazelcast.diagnostics.enabled=true to the JVM arguments.
Feb 28, 2018 9:47:17 PM com.hazelcast.core.LifecycleService
INFO: [172.16.18.169]:5701 [dev] [3.10-BETA-1] [172.16.18.169]:5701 is STARTING
Feb 28, 2018 9:47:18 PM com.hazelcast.nio.tcp.TcpIpConnector
INFO: [172.16.18.169]:5701 [dev] [3.10-BETA-1] Connecting to /10.253.67.76:5701, timeout: 0, bind-any: true
Feb 28, 2018 9:47:18 PM com.hazelcast.nio.tcp.TcpIpConnector
INFO: [172.16.18.169]:5701 [dev] [3.10-BETA-1] Connecting to /172.16.17.122:5701, timeout: 0, bind-any: true
Feb 28, 2018 9:47:18 PM com.hazelcast.nio.tcp.TcpIpConnector
INFO: [172.16.18.169]:5701 [dev] [3.10-BETA-1] Connecting to /172.16.13.23:5701, timeout: 0, bind-any: true
Feb 28, 2018 9:47:18 PM com.hazelcast.nio.tcp.TcpIpConnector
INFO: [172.16.18.169]:5701 [dev] [3.10-BETA-1] Connecting to /172.16.60.55:5701, timeout: 0, bind-any: true
Feb 28, 2018 9:47:18 PM com.hazelcast.nio.tcp.TcpIpConnector
INFO: [172.16.18.169]:5701 [dev] [3.10-BETA-1] Could not connect to: /172.16.17.122:5701. Reason: SocketException[Connection refused to address /172.16.17.122:5701]
Feb 28, 2018 9:47:18 PM com.hazelcast.nio.tcp.TcpIpConnector
INFO: [172.16.18.169]:5701 [dev] [3.10-BETA-1] Could not connect to: /172.16.13.23:5701. Reason: SocketException[Connection refused to address /172.16.13.23:5701]
Feb 28, 2018 9:47:18 PM com.hazelcast.nio.tcp.TcpIpConnector
INFO: [172.16.18.169]:5701 [dev] [3.10-BETA-1] Could not connect to: /172.16.60.55:5701. Reason: SocketException[Connection refused to address /172.16.60.55:5701]
Feb 28, 2018 9:47:18 PM com.hazelcast.internal.cluster.impl.DiscoveryJoiner
INFO: [172.16.18.169]:5701 [dev] [3.10-BETA-1] [172.16.17.122]:5701 is added to the blacklist.
Feb 28, 2018 9:47:18 PM com.hazelcast.internal.cluster.impl.DiscoveryJoiner
INFO: [172.16.18.169]:5701 [dev] [3.10-BETA-1] [172.16.60.55]:5701 is added to the blacklist.
Feb 28, 2018 9:47:18 PM com.hazelcast.internal.cluster.impl.DiscoveryJoiner
INFO: [172.16.18.169]:5701 [dev] [3.10-BETA-1] [172.16.13.23]:5701 is added to the blacklist.
Feb 28, 2018 9:47:23 PM com.hazelcast.system
INFO: [172.16.18.169]:5701 [dev] [3.10-BETA-1] Cluster version set to 3.10
Feb 28, 2018 9:47:23 PM com.hazelcast.internal.cluster.ClusterService
INFO: [172.16.18.169]:5701 [dev] [3.10-BETA-1]
Members {size:1, ver:1} [
Member [172.16.18.169]:5701 - 1787f7b3-afbb-4dab-a375-c9d55f5ecbcf this
]
Feb 28, 2018 9:47:23 PM com.hazelcast.core.LifecycleService
INFO: [172.16.18.169]:5701 [dev] [3.10-BETA-1] [172.16.18.169]:5701 is STARTED
Token retrieval from file not working
when trying to use the REST API method, it fails to read the token from file, however also doesnt throw an exception that reading fails.
all you see is a null for the token
INFO: [10.4.1.222]:5701 [dev] [3.7.4] Kubernetes Discovery properties: { service-dns: null, service-dns-timeout: 5, service-name: null, service-label: vertx-cluster, service-label-value: true, namespace: production, kubernetes-master: https://kubernetes.default.svc}
Sep 06, 2017 1:44:03 PM com.hazelcast.spi.discovery.integration.DiscoveryService
INFO: [10.4.1.222]:5701 [dev] [3.7.4] Kubernetes Discovery: Bearer Token { null }
and later
SEVERE: [10.4.1.222]:5701 [dev] [3.7.4] null
java.lang.NullPointerException
at com.hazelcast.kubernetes.ServiceEndpointResolver.getSimpleDiscoveryNodes(ServiceEndpointResolver.java:115)
at com.hazelcast.kubernetes.ServiceEndpointResolver.getDiscoveryNodes(ServiceEndpointResolver.java:105)
at com.hazelcast.kubernetes.ServiceEndpointResolver.resolve(ServiceEndpointResolver.java:84)
at com.hazelcast.kubernetes.HazelcastKubernetesDiscoveryStrategy.discoverNodes(HazelcastKubernetesDiscoveryStrategy.java:98)
if you specify the token via property api-token , its all working fine
PS: DNS method didnt work at all but no errors
Unable to connect client using kubernetes discovery strategy from xml
I have a hazelcast 3.9.2 running on kubernetes which is using the HazelcastKubernetesDiscoveryStrategy, no issues.
When I try to connect a client using the following xml config.
<hazelcast-client>
<properties>
<property name="hazelcast.discovery.enabled">true</property>
</properties>
<network>
<discovery-strategies>
<discovery-strategy enabled="true"
class="com.hazelcast.kubernetes.HazelcastKubernetesDiscoveryStrategy">
<properties>
<property name="service-name">something</property>
<property name="namespace">something</property>
</properties>
</discovery-strategy>
</discovery-strategies>
</network>
</hazelcast-client>I get the following error.
com.hazelcast.config.InvalidConfigurationException: Invalid configuration
at com.hazelcast.spi.discovery.impl.DefaultDiscoveryService.loadDiscoveryStrategies(DefaultDiscoveryService.java:151)
at com.hazelcast.spi.discovery.impl.DefaultDiscoveryService.<init>(DefaultDiscoveryService.java:60)
at com.hazelcast.spi.discovery.impl.DefaultDiscoveryServiceProvider.newDiscoveryService(DefaultDiscoveryServiceProvider.java:29)
at com.hazelcast.client.impl.HazelcastClientInstanceImpl.initDiscoveryService(HazelcastClientInstanceImpl.java:330)
at com.hazelcast.client.impl.HazelcastClientInstanceImpl.<init>(HazelcastClientInstanceImpl.java:239)
at com.hazelcast.client.HazelcastClient$1.createHazelcastInstanceClient(HazelcastClient.java:55)
at com.hazelcast.client.HazelcastClientManager.newHazelcastClient(HazelcastClientManager.java:74)
at com.hazelcast.client.HazelcastClient.newHazelcastClient(HazelcastClient.java:72)
at CacheConsumer.runApp(CacheConsumer.java:49)
at CacheConsumer.main(CacheConsumer.java:26)
Caused by: com.hazelcast.config.properties.ValidationException: There is no discovery strategy factory to create 'DiscoveryStrategyConfig{properties={namespace=aws-hazelcast, service-name=hazelcast}, className='com.hazelcast.kubernetes.HazelcastKubernetesDiscoveryStrategy', discoveryStrategyFactory=null}' Is it a typo in a strategy classname? Perhaps you forgot to include implementation on a classpath?
at com.hazelcast.spi.discovery.impl.DefaultDiscoveryService.buildDiscoveryStrategy(DefaultDiscoveryService.java:188)
at com.hazelcast.spi.discovery.impl.DefaultDiscoveryService.loadDiscoveryStrategies(DefaultDiscoveryService.java:145)But when I try the following code it works fine.
DiscoveryConfig dc = config.getNetworkConfig().getDiscoveryConfig();
HazelcastKubernetesDiscoveryStrategyFactory factory = new HazelcastKubernetesDiscoveryStrategyFactory();
DiscoveryStrategyConfig strategyConfig = new DiscoveryStrategyConfig(factory);
strategyConfig.addProperty("service-name","something");
strategyConfig.addProperty("namespace","something");
dc.addDiscoveryStrategyConfig(strategyConfig);There was a similar issue opened but closed:
#59
The issue suggested possible class path issue, it doesn't seem like a class path issue.
The error message has:
discoveryStrategyFactory=nullJavadoc discrepancy
Hi!
There is minor discrepancy in property definition (missed dash in resolve-not-ready-addresses) between the javadoc and the code:
and
Performance degradation seen using plugin
I see a performance degradation using plugin comparing to use a standalone mode of Hazelcast. Degradation is about 2x-3x slower.
My server side config:
Config config = com.hazelcast.config.Config();
config.setProperty("hazelcast.discovery.enabled", "true");
NetworkConfig networkConfig = config.getNetworkConfig();
networkConfig.getJoin().getMulticastConfig().setEnabled(false);
networkConfig.getJoin().getTcpIpConfig().setEnabled(false);
DiscoveryStrategyConfig discoveryStrategyConfig = new DiscoveryStrategyConfig(new HazelcastKubernetesDiscoveryStrategyFactory());
String serviceName = "hazelcast";
discoveryStrategyConfig.addProperty("service-name", serviceName);
DiscoveryConfig discoveryConfig = new DiscoveryConfig();
discoveryConfig.addDiscoveryStrategyConfig(discoveryStrategyConfig);
networkConfig.getJoin().setDiscoveryConfig(discoveryConfig);
My client side config:
ClientConfig clientConfig = new ClientConfig();
ClientNetworkConfig networkConfigCloud = new ClientNetworkConfig();
clientConfig.setProperty("hazelcast.discovery.enabled", "true");
DiscoveryStrategyConfig discoveryStrategyConfig = new DiscoveryStrategyConfig(new HazelcastKubernetesDiscoveryStrategyFactory());
String serviceName = "hazelcast";
discoveryStrategyConfig.addProperty("service-name", serviceName);
DiscoveryConfig discoveryConfig = new DiscoveryConfig();
discoveryConfig.addDiscoveryStrategyConfig(discoveryStrategyConfig);
networkConfigCloud.setDiscoveryConfig(discoveryConfig);
Some parameters:
- Service name called "hazelcast" is a Kubernetes service.
- Using Hazelcast 3.6.
- Using Kubernetes 1.5.1.
Am I missing something? What could be the issue?
Thanks in advance.
Add retries for node discovery
The Kubernetes REST API should be called with retries. Otherwise, there is a risk of a sudden failure just because Kubernetes Master is down.
User "system:serviceaccount:poc:default" cannot get endpoints in project
User "system:serviceaccount:poc:default" cannot get endpoints in project ****, do we need to create a new sa for deploying application?
NoSuchMethodError: io.fabric8.kubernetes.client.ConfigBuilder.build()Lio/fabric8/kubernetes/client/Config;
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [io.fabric8.kubernetes.client.Config]: Factory method 'kubernetesClientConfig' threw exception; nested exception is java.lang.NoSuchMethodError: io.fabric8.kubernetes.client.ConfigBuilder.build()Lio/fabric8/kubernetes/client/Config;
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:189)
at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:588)
... 23 common frames omitted
Caused by: java.lang.NoSuchMethodError: io.fabric8.kubernetes.client.ConfigBuilder.build()Lio/fabric8/kubernetes/client/Config;
Not possible to only provide a service-dns
According to the README it should be possible to provide only a service-dns property when DNS based lookups are required.
However if only this property is provided then it fails to load properties throwing an exception:
"For kubernetes discovery either 'service-dns' or " + "'service-name' and 'namespace' must be set");
"
To fix, the line should read:
if (serviceDns == null && (serviceName == null || namespace == null)) {
Discovery issue when doing blue green rolling updates
Hello team,
I'm not sure if this is the correct place for this issue, but I believe its atleast a starting point, so please do let me know if I was wrong to create it here.
We use Hazelcast as 2nd level cache for Hibernate and I'm facing a weird issue, during a blue-green deployment scenario below with Kubernetes and I doubt if it is due to the discovery strategy, somehow I have never faced a similar issue with Hazelcast in the past with multicast and with Eureka discovery as we use it extensively in JHipster.
- The application is clustered and runs in Pod A and Pod B with a load balancer
- The application is at V1 for example
- We deploy V2 of application, which has new fields added to an entity class, to Pod A
- V2 successfully deploys in Pod A and updates the DB with new schema changes
- Somehow app in Pod B with V1 replicates its cache to Pod A which is in V2
- Now we deploy Pod B with V2 successfully
- Now app in Pod A which has V2 of the app but incorrect cache(v1) replicates to Pod B and hence both pod has old cache data
- Now when you access the entity Hibernate fails with below stack trace
Can this be solved in the discovery strategy so that newer versions do not try to replicate from an older version of the app? Is my understanding wrong?
The below is the stack trace from Hibernate when it reads from the cache as the cache is outdated now
java.lang.ArrayIndexOutOfBoundsException: 11
at org.hibernate.type.TypeHelper.assemble(TypeHelper.java:96)
at org.hibernate.cache.spi.entry.StandardCacheEntryImpl.assemble(StandardCacheEntryImpl.java:141)
at org.hibernate.event.internal.DefaultLoadEventListener.convertCacheEntryToEntity(DefaultLoadEventListener.java:754)
at org.hibernate.event.internal.DefaultLoadEventListener.processCachedEntry(DefaultLoadEventListener.java:625)
at org.hibernate.event.internal.DefaultLoadEventListener.loadFromSecondLevelCache(DefaultLoadEventListener.java:602)
at org.hibernate.event.internal.DefaultLoadEventListener.doLoad(DefaultLoadEventListener.java:462)
at org.hibernate.event.internal.DefaultLoadEventListener.load(DefaultLoadEventListener.java:219)
at org.hibernate.event.internal.DefaultLoadEventListener.proxyOrLoad(DefaultLoadEventListener.java:278)
at org.hibernate.event.internal.DefaultLoadEventListener.doOnLoad(DefaultLoadEventListener.java:121)
at org.hibernate.event.internal.DefaultLoadEventListener.onLoad(DefaultLoadEventListener.java:89)
at org.hibernate.internal.SessionImpl.fireLoad(SessionImpl.java:1237)
at org.hibernate.internal.SessionImpl.internalLoad(SessionImpl.java:1120)
at org.hibernate.type.EntityType.resolveIdentifier(EntityType.java:646)
at org.hibernate.type.EntityType.resolve(EntityType.java:431)
at org.hibernate.engine.internal.TwoPhaseLoad.doInitializeEntity(TwoPhaseLoad.java:165)
at org.hibernate.engine.internal.TwoPhaseLoad.initializeEntity(TwoPhaseLoad.java:125)
at org.hibernate.loader.Loader.initializeEntitiesAndCollections(Loader.java:1146)
at org.hibernate.loader.Loader.processResultSet(Loader.java:1005)
at org.hibernate.loader.Loader.doQuery(Loader.java:943)
at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:349)
at org.hibernate.loader.Loader.doList(Loader.java:2615)
at org.hibernate.loader.Loader.doList(Loader.java:2598)
at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2430)
at org.hibernate.loader.Loader.list(Loader.java:2425)
at org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:502)
at org.hibernate.hql.internal.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:379)
at org.hibernate.engine.query.spi.HQLQueryPlan.performList(HQLQueryPlan.java:216)
at org.hibernate.internal.SessionImpl.list(SessionImpl.java:1488)
at org.hibernate.query.internal.AbstractProducedQuery.doList(AbstractProducedQuery.java:1445)
at org.hibernate.query.internal.AbstractProducedQuery.list(AbstractProducedQuery.java:1414)
at org.hibernate.query.Query.getResultList(Query.java:146)
at org.hibernate.query.criteria.internal.compile.CriteriaQueryTypeQueryAdapter.getResultList(CriteriaQueryTypeQueryAdapter.java:72)
at org.springframework.data.jpa.repository.support.SimpleJpaRepository.findAll(SimpleJpaRepository.java:329)
at org.springframework.data.jpa.repository.support.SimpleJpaRepository.findAll(SimpleJpaRepository.java:74)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.data.repository.core.support.RepositoryFactorySupport$QueryExecutorMethodInterceptor.executeMethodOn(RepositoryFactorySupport.java:504)
at org.springframework.data.repository.core.support.RepositoryFactorySupport$QueryExecutorMethodInterceptor.doInvoke(RepositoryFactorySupport.java:489)
at org.springframework.data.repository.core.support.RepositoryFactorySupport$QueryExecutorMethodInterceptor.invoke(RepositoryFactorySupport.java:461)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.data.projection.DefaultMethodInvokingMethodInterceptor.invoke(DefaultMethodInvokingMethodInterceptor.java:56)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:282)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:136)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.data.jpa.repository.support.CrudMethodMetadataPostProcessor$CrudMethodMetadataPopulatingMethodInterceptor.invoke(CrudMethodMetadataPostProcessor.java:133)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.data.repository.core.support.SurroundingTransactionDetectorMethodInterceptor.invoke(SurroundingTransactionDetectorMethodInterceptor.java:57)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
at com.sun.proxy.$Proxy183.findAll(Unknown Source)
at com.xxx.xxx.service.DashboardService.findAll(DashboardService.java:89)
at com.xxx.xxx.service.DashboardService$$FastClassBySpringCGLIB$$4c9a082d.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:282)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673)
at com.xxx.xxx.service.DashboardService$$EnhancerBySpringCGLIB$$6bdfcfa9.findAll(<generated>)
at com.xebialabs.agatha.web.rest.ProjectDashboardResource.getAllProjectDashboards(ProjectDashboardResource.java:95)
at com.xebialabs.agatha.web.rest.ProjectDashboardResource$$FastClassBySpringCGLIB$$180b3b55.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at com.ryantenney.metrics.spring.TimedMethodInterceptor.invoke(TimedMethodInterceptor.java:48)
at com.ryantenney.metrics.spring.TimedMethodInterceptor.invoke(TimedMethodInterceptor.java:34)
at com.ryantenney.metrics.spring.AbstractMetricMethodInterceptor.invoke(AbstractMetricMethodInterceptor.java:59)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673)
at com.xebialabs.agatha.web.rest.ProjectDashboardResource$$EnhancerBySpringCGLIB$$bb71fab6.getAllProjectDashboards(<generated>)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129)
at com.codahale.metrics.servlet.AbstractInstrumentedFilter.doFilter(AbstractInstrumentedFilter.java:111)
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
at org.springframework.boot.web.filter.ApplicationContextHeaderFilter.doFilterInternal(ApplicationContextHeaderFilter.java:55)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:101)
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
at org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:110)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:114)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:170)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at com.xebialabs.agatha.security.jwt.JWTFilter.doFilter(JWTFilter.java:43)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.web.filter.CorsFilter.doFilterInternal(CorsFilter.java:96)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:116)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:64)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.access.channel.ChannelProcessingFilter.doFilter(ChannelProcessingFilter.java:157)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:262)
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:108)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:81)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)
at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
at io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:64)
at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131)
at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undertow.security.handlers.AuthenticationConstraintHandler.handleRequest(AuthenticationConstraintHandler.java:53)
at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
at io.undertow.servlet.handlers.security.ServletSecurityConstraintHandler.handleRequest(ServletSecurityConstraintHandler.java:59)
at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292)
at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
at io.undertow.server.Connectors.executeRootHandler(Connectors.java:332)
at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:812)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)Could hazlecast-kuberntes work with 3.6.5?
Hi, sorry for the question posted here.
I'm working to deploy OrientDB 2.2.x on Google Containers.
ODB uses HZ 3.6.5, so I recompiled this project for it without any error in the build.
At runtime the com.hazelcast.HazelcastKubernetesDiscoveryStrategy isn't loaded.
Maybe I need to modify my configuration file or to modify the code to let it work with HZ 3.6.5
thank you in advance for the support
Make Kubernetes ZONE_AWARE
Use the label value of failure-domain.beta.kubernetes.io/zone to retrieve the zone and make Hazelcast Zone Aware.
The label can be retrieved as follows from Kubernetes Master:
- Get
nodeNamefrom/api/v1/namespaces/<namespace>/endpoints - Get label value from
/api/v1/nodes/<nodeName>
Note that the RBAC sample yaml (+ Helm Charts) need to be adapted to give read access to the /nodes endpoint.
Incorrect PORT from SRV record
I am running a kubernetes cluster with skyDNS service on
The hazelcast container logs indicate that it is trying to connect on port :0
Nov 15, 2016 10:35:11 PM com.hazelcast.nio.tcp.InitConnectionTask INFO: [11.11.104.3]:5701 [dev] [3.6.1] Connecting to /11.11.109.2:0, timeout: 0, bind-any: true Nov 15, 2016 10:35:11 PM com.hazelcast.nio.tcp.InitConnectionTask INFO: [11.11.104.3]:5701 [dev] [3.6.1] Connecting to /11.11.104.3:0, timeout: 0, bind-any: true Nov 15, 2016 10:35:11 PM com.hazelcast.nio.tcp.InitConnectionTask INFO: [11.11.104.3]:5701 [dev] [3.6.1] Could not connect to: /11.11.104.3:0. Reason: SocketException[Connection refused to address /11.11.104.3:0] Nov 15, 2016 10:35:11 PM com.hazelcast.cluster.impl.DiscoveryJoiner INFO: [11.11.104.3]:5701 [dev] [3.6.1] Address[11.11.104.3]:0 is added to the blacklist. Nov 15, 2016 10:35:11 PM com.hazelcast.nio.tcp.InitConnectionTask INFO: [11.11.104.3]:5701 [dev] [3.6.1] Could not connect to: /11.11.109.2:0. Reason: SocketException[Connection refused to address /11.11.109.2:0] Nov 15, 2016 10:35:11 PM com.hazelcast.cluster.impl.DiscoveryJoiner INFO: [11.11.104.3]:5701 [dev] [3.6.1] Address[11.11.109.2]:0 is added to the blacklist. Nov 15, 2016 10:35:12 PM com.hazelcast.cluster.impl.DiscoveryJoiner
I have created a headless kubernetes service for hazelcast and nslookup returns
ip addresses of all 3 hazelcast pods
nslookup hazelcast.kube-infra.svc.cluster.local 172.16.48.2
Server: 172.16.48.2
Address 1: 172.16.48.2 kube-dns.kube-system.svc.cluster.local
Name: hazelcast.kube-infra.svc.cluster.local
Address 1: 11.11.38.2
Address 2: 11.11.109.2
Address 3: 11.11.104.3
I am wondering how to make it connect to 5701 (which is the port where hazelcast is running in all pods)
kubectl describe svc hazelcast --namespace=kube-infra
Name: hazelcast
Namespace: kube-infra
Labels: app=hazelcast
version=latest
Selector: app=hazelcast,version=latest
Type: ClusterIP
IP: None
Port: tcp-5701-5701-nfj29 5701/TCP
Endpoints: 11.11.104.3:5701,11.11.109.2:5701,11.11.38.2:5701
Session Affinity: None
Implement retry pattern for REST API calls
Sometimes we get the following exception under an heavy load context:
Error executing: GET at: https://kubernetes.default.svc/api/v1/namespaces/default/endpoints/hazelcast. Cause: Failed to connect to kubernetes.default.svc/10.239.240.1:443 , io.fabric8.kubernetes.client.KubernetesClientException: Error executing: GET at: https://kubernetes.default.svc/api/v1/namespaces/default/endpoints/hazelcast. Cause: Failed to connect to kubernetes.default.svc/10.239.240.1:443 at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestException(OperationSupport.java:272
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:205
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:196
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleGet(BaseOperation.java:483
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.get(BaseOperation.java:108
at com.noctarius.hazelcast.kubernetes.ServiceEndpointResolver.resolve(ServiceEndpointResolver.java:71
at com.noctarius.hazelcast.kubernetes.HazelcastKubernetesDiscoveryStrategy.discoverNodes(HazelcastKubernetesDiscoveryStrategy.java:88
at com.hazelcast.spi.discovery.impl.DefaultDiscoveryService.discoverNodes(DefaultDiscoveryService.java:74
at com.hazelcast.client.spi.impl.discovery.DiscoveryAddressTranslator.refresh(DiscoveryAddressTranslator.java:67
at com.hazelcast.client.spi.impl.discovery.DiscoveryAddressTranslator.translate(DiscoveryAddressTranslator.java:56
at com.hazelcast.client.connection.nio.ClientConnectionManagerImpl.getOrTriggerConnect(ClientConnectionManagerImpl.java:227
at com.hazelcast.client.connection.nio.ClientConnectionManagerImpl.getOrTriggerConnect(ClientConnectionManagerImpl.java:67
at com.hazelcast.client.spi.impl.ClientSmartInvocationServiceImpl.getConnection(ClientSmartInvocationServiceImpl.java:78
at com.hazelcast.client.spi.impl.ClientSmartInvocationServiceImpl.invokeOnPartitionOwner(ClientSmartInvocationServiceImpl.java:49
at com.hazelcast.client.spi.impl.ClientInvocation.invokeOnSelection(ClientInvocation.java:152
at com.hazelcast.client.spi.impl.ClientInvocation.invoke(ClientInvocation.java:132
at com.hazelcast.client.spi.ClientProxy.invokeOnPartition(ClientProxy.java:125
at com.hazelcast.client.spi.ClientProxy.invoke(ClientProxy.java:120
at com.hazelcast.client.proxy.ClientMapProxy.tryLock(ClientMapProxy.java:561
at com.hazelcast.client.proxy.ClientMapProxy.tryLock(ClientMapProxy.java:549
at com.idea2.distributed.Locks.lock(Locks.java:57
at com.idea2.distributed.Locks.lock(Locks.java:39
at com.idea2.distributed.Locks.lock(Locks.java:31
at com.idea2.shared.logic.integrations.IntegrationsManager.privateDeployEndpoint(IntegrationsManager.java:333
at com.idea2.shared.logic.integrations.IntegrationsManager.restartEndpoint(IntegrationsManager.java:546
at com.idea2.shared.logic.integrations.IntegrationsManager.restartEndpoint(IntegrationsManager.java:493
at com.idea2.apps.logic.refactoring.DataRefactoringService.applyEndpointChange(DataRefactoringService.java:417
at com.idea2.apps.logic.refactoring.DataRefactoringService.propagateChange(DataRefactoringService.java:406
at com.idea2.apps.logic.refactoring.DataRefactoringService.applyDataChanges(DataRefactoringService.java:290
at com.idea2.apps.logic.refactoring.DataRefactoringService.applyChanges(DataRefactoringService.java:261
at com.idea2.apps.logic.jobs.PushChangesUpdateDataJobExecutor.executeChunk(PushChangesUpdateDataJobExecutor.java:46
at com.idea2.apps.logic.jobs.framework.BackgroundJobExecutor.wrapExecuteChunk(BackgroundJobExecutor.java:86
at com.idea2.apps.logic.jobs.AppsJobsExecutor.executeChunk(AppsJobsExecutor.java:78
at com.idea2.apps.logic.jobs.framework.BackgroundJobChunkExecutionThread.run(BackgroundJobChunkExecutionThread.java:36
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617
at java.lang.Thread.run(Thread.java:745
at ------ End remote and begin local stack-trace ------.(Unknown Source
at com.hazelcast.client.spi.impl.ClientInvocationFuture.resolveResponse(ClientInvocationFuture.java:133
at com.hazelcast.client.spi.impl.ClientInvocationFuture.get(ClientInvocationFuture.java:100
at com.hazelcast.client.spi.impl.ClientInvocationFuture.get(ClientInvocationFuture.java:75
at com.hazelcast.client.spi.impl.ClientInvocationFuture.get(ClientInvocationFuture.java:38
at com.hazelcast.client.spi.ClientProxy.invokeOnPartition(ClientProxy.java:126
at com.hazelcast.client.spi.ClientProxy.invoke(ClientProxy.java:120
at com.hazelcast.client.proxy.ClientMapProxy.tryLock(ClientMapProxy.java:561
at com.hazelcast.client.proxy.ClientMapProxy.tryLock(ClientMapProxy.java:549
at com.idea2.distributed.Locks.lock(Locks.java:57
at com.idea2.distributed.Locks.lock(Locks.java:39
at com.idea2.distributed.Locks.lock(Locks.java:31
at com.idea2.shared.logic.integrations.IntegrationsManager.privateDeployEndpoint(IntegrationsManager.java:333
at com.idea2.shared.logic.integrations.IntegrationsManager.restartEndpoint(IntegrationsManager.java:546
at com.idea2.shared.logic.integrations.IntegrationsManager.restartEndpoint(IntegrationsManager.java:493
at com.idea2.apps.logic.refactoring.DataRefactoringService.applyEndpointChange(DataRefactoringService.java:417
at com.idea2.apps.logic.refactoring.DataRefactoringService.propagateChange(DataRefactoringService.java:406
at com.idea2.apps.logic.refactoring.DataRefactoringService.applyDataChanges(DataRefactoringService.java:290
at com.idea2.apps.logic.refactoring.DataRefactoringService.applyChanges(DataRefactoringService.java:261
at com.idea2.apps.logic.jobs.PushChangesUpdateDataJobExecutor.executeChunk(PushChangesUpdateDataJobExecutor.java:46
at com.idea2.apps.logic.jobs.framework.BackgroundJobExecutor.wrapExecuteChunk(BackgroundJobExecutor.java:86
at com.idea2.apps.logic.jobs.AppsJobsExecutor.executeChunk(AppsJobsExecutor.java:78
at com.idea2.apps.logic.jobs.framework.BackgroundJobChunkExecutionThread.run(BackgroundJobChunkExecutionThread.java:36
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617
at java.lang.Thread.run(Thread.java:745) Caused by: java.net.ConnectException: Failed to connect to kubernetes.default.svc/10.239.240.1:443 at com.squareup.okhttp.internal.io.RealConnection.connectSocket(RealConnection.java:139
at com.squareup.okhttp.internal.io.RealConnection.connect(RealConnection.java:108
at com.squareup.okhttp.internal.http.StreamAllocation.findConnection(StreamAllocation.java:174
at com.squareup.okhttp.internal.http.StreamAllocation.findHealthyConnection(StreamAllocation.java:126
at com.squareup.okhttp.internal.http.StreamAllocation.newStream(StreamAllocation.java:95
at com.squareup.okhttp.internal.http.HttpEngine.connect(HttpEngine.java:283
at com.squareup.okhttp.internal.http.HttpEngine.sendRequest(HttpEngine.java:224
at com.squareup.okhttp.Call.getResponse(Call.java:286
at com.squareup.okhttp.Call$ApplicationInterceptorChain.proceed(Call.java:243
at io.fabric8.kubernetes.client.utils.HttpClientUtils$3.intercept(HttpClientUtils.java:110
at com.squareup.okhttp.Call$ApplicationInterceptorChain.proceed(Call.java:232
at com.squareup.okhttp.Call.getResponseWithInterceptorChain(Call.java:205
at com.squareup.okhttp.Call.execute(Call.java:80
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:203
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:196
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleGet(BaseOperation.java:483
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.get(BaseOperation.java Context
Update kubernetes-client version
HI,
when using vertx application on openshift (vertx uses hazelcast as cluster manager) I get a conflict with the kubernetes-client version as this hazelcast-kubernetes project still uses old version 1.3.66 while the vertx-config-kubernetes-configmap uses version 1.4.35. Having both installed the configmap module doesnt work.
Nodes/Instances are registered more than once
Hi,
i build a small spring boot app (https://github.com/codeCoder/spring-boot-hello-world)
to get involved with kubernetes and hazelcast in it.
During the start it looks like a single hazelcast instance is registered more than once in the cluster with different ports. In https://github.com/codeCoder/spring-boot-hello-world/blob/master/hello_world_last_run_first_pod
you see following lines
Members {size:3, ver:3} [
Member [172.17.0.8]:5701 - 96d9f638-9a22-4305-a68c-4e222b10e4aa this
Member [172.17.0.8]:5702 - 065b6ec2-3996-4230-a186-ebc4e5d6a3a5
Member [172.17.0.8]:5703 - b087dfc0-bd6b-4e9e-b0a7-44dd324e8034
]
If i start a second pod(see: https://github.com/codeCoder/spring-boot-hello-world/blob/master/hello_world_last_run_second_pod)
you see much more nodes/ instances registered in the cluster:
Members {size:6, ver:6} [
Member [172.17.0.8]:5701 - 96d9f638-9a22-4305-a68c-4e222b10e4aa
Member [172.17.0.8]:5702 - 065b6ec2-3996-4230-a186-ebc4e5d6a3a5
Member [172.17.0.8]:5703 - b087dfc0-bd6b-4e9e-b0a7-44dd324e8034
Member [172.17.0.11]:5701 - a8bc9171-e10f-4c26-9ad6-1a6fc43446ce
Member [172.17.0.11]:5702 - 1669f5a6-1fa5-442b-9745-c2e7a014be03 this
Member [172.17.0.11]:5703 - c41ac3df-f2d1-4ef7-88ca-8cf0afd78b54
]
At the moment I use this rest config: https://github.com/codeCoder/spring-boot-hello-world/blob/master/src/main/resources/hazelcast-kubernetes-restapi.xml
with following network settings:
<port>5701</port>
<join>
<!-- deactivate normal discovery -->
<multicast enabled="false"/>
<tcp-ip enabled="false"/>
<!-- activate the Kubernetes plugin -->
<discovery-strategies>
<discovery-strategy enabled="true"
class="com.hazelcast.kubernetes.HazelcastKubernetesDiscoveryStrategy">
<properties>
<!-- configure discovery service API lookup -->
<property name="service-name">spring-boot-hello-world-service-cluster</property>
<property name="service-label-name">run</property>
<property name="service-label-value">spring-boot-hello-world-service-cluster</property>
<property name="namespace">default</property>
</properties>
</discovery-strategy>
</discovery-strategies>
</join>
</network>
I still have to test if the cache is working properly. But first I would like to understand this behavior.
Write Blog Post about using Kubernetes
Write a Blog Post or a series of blog posts about:
- Using Kubernetes plugin
- with DNS Lookup Discovery
- with REST API Discovery
- Using Hazelcast Helm Chart
Kubernetes [1] and OpenShift [2] code samples can be used.
[1] https://github.com/hazelcast/hazelcast-code-samples/tree/master/hazelcast-integration/kubernetes
[2] https://github.com/hazelcast/hazelcast-code-samples/tree/master/hazelcast-integration/openshift
Update minimal-json dependencies
Minimal JSON library package has changed from: ""com.hazelcast.com.eclipsesource.json" (3.10) to "com.hazelcast.internal.json" (3.11). It needs to be updated.
Update copyright license and code formatting
The license in files is outdates (2017).
endpoints.getSubsets() should be checked for null
endpoints.getSubsets() in getSimpleDiscoveryNodes may return null in some circumstances.
As we don't check for null, this would result in an NPE being thrown.
See #42 (comment)
Is it neccesary to configure discovery on client side when using a Kubernetes service?
If clients connect to server through a Kubernetes service, Is it neccesary to configure discovery on client side?
Enable external Smart Client (expose PODs and assign public IP automatically using Kubernetes API)
Currently, if someone deploys a Hazelcast cluster (manually or via Helm Chart), it's accessible via LoadBalancer service and therefore available only with the use of Dummy Client. That is the standard way how Kubernetes deals with multiple PODs. They are accessible only via one service [1].
It should be possible to expose each POD separately (using the Kubernetes Adds-On "Service Per POD" [2]). Then, we could use Kubernetes API to get the information about external IPs and set it in discoverNodes() or AddressPicker implementation.
[1] https://stackoverflow.com/questions/45650526/kubernetes-expose-only-single-pod-of-statefulset
[2] https://github.com/GoogleCloudPlatform/metacontroller/tree/master/examples/service-per-pod
Connection failed to Kubernetes when using the service strategy
When Kubernetes runs without the DNS support (as it's a plugin), the resolution fails.
The host is unknown. In this case the URL can be computed as follows: KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT}
Revise readme
The main readme of this project could get some love. It talks too much about implementation details and too little about actual usage.
Example - this is close to the top of the readme:
While this is at the bottom:
The SPI is purely an implementation detail, while I reckon majority of users are interested in the Docker images.
In general the readme should be more use-case oriented.
New release?
Any chance of a release please?
PROPERTY_DEFINITIONS is missing SERVICE_DNS_TIMEOUT in HazelcastKubernetesDiscoveryStrategyFactory.java
Right after this line is where the property should be declared...
Instruction doesn't applied for Hazelcast 3.9.1
For the Hazelcast 3.9.1
I've tried with with configuration
<!-- activate the Kubernetes plugin -->
<discovery-strategies>
<discovery-strategy enabled="true"
class="com.hazelcast.kubernetes.HazelcastKubernetesDiscoveryStrategy">
<properties>
<!-- configure discovery service API lookup -->
<property name="service-name">MY-SERVICE-NAME</property>
<property name="service-label-name">cluster01</property>
<property name="service-label-value">true</property>
<property name="namespace">MY-KUBERNETES-NAMESPACE</property>
</properties>
</discovery-strategy>
</discovery-strategies>
But I then encounter an error shown as below:
Caused by: com.hazelcast.config.properties.ValidationException: There is no discovery strategy factory to create 'DiscoveryStrategyConfig{properties={namespace=default}, className='com.hazelcast.kubernetes.HazelcastKubernetesDiscoveryStrategy', discoveryStrategyFactory=null}' Is it a typo in a strategy classname? Perhaps you forgot to include implementation on a classpath?
at com.hazelcast.spi.discovery.impl.DefaultDiscoveryService.buildDiscoveryStrategy(DefaultDiscoveryService.java:188)
at com.hazelcast.spi.discovery.impl.DefaultDiscoveryService.loadDiscoveryStrategies(DefaultDiscoveryService.java:145)
Property names in the README are not correct
They need to be prefixed with:
hazelcast.kubernetes. in order to work
Automatic discovery of all nodes in namespace causes trouble with 2 or more separate HZ clusters
Hi,
with 0.9.3 version I saw that resolve() was made "smarter" (https://github.com/noctarius/hazelcast-kubernetes-discovery/blob/master/src/main/java/com/noctarius/hazelcast/kubernetes/ServiceEndpointResolver.java#L68). But in my case that is quite a problem.
We have two separate hazelcast clusters on kubernetes (in same namespace). Let's say I start one of them. Two pods come up and I'd expect they will eventually form a cluster (but until hz is intialized readiness check prevents one to see the other node). That means I expect the discovery plugin to say there are no other nodes. What happens instead is that nodes from the other cluster are discovered but credentials are incorrect so at the end hz can't connect anywhere and just terminates.
2016-11-11 13:40:13,247 [localhost-startStop-1] [DEBUG] [c.h.i.c.impl.DiscoveryJoiner] - [10.201.36.12]:5701 [mts-dev-ts-cluster] [3.7.2] PostJoin master: null, isMaster: false
2016-11-11 13:40:13,248 [localhost-startStop-1] [ERROR] [com.hazelcast.instance.Node] - [10.201.36.12]:5701 [mts-dev-ts-cluster] [3.7.2] Could not join cluster in 300000 ms. Shutting down now!
2016-11-11 13:40:13,250 [localhost-startStop-1] [INFO ] [c.h.core.LifecycleService] - [10.201.36.12]:5701 [mts-dev-ts-cluster] [3.7.2] [10.201.36.12]:5701 is SHUTTING_DOWN
2016-11-11 13:40:13,251 [localhost-startStop-1] [WARN ] [com.hazelcast.instance.Node] - [10.201.36.12]:5701 [mts-dev-ts-cluster] [3.7.2] Terminating forcefully...
Can this additional logic to discover nodes be disabled somehow?
"Wrong bind request from..." message on Hazelcast startup in Google's Kubernetes
First, thanks so much for writing and maintaining this project, it's exactly what I need.
So I've written a simple Java app that uses a Hazelcast Lock to ensure that a task only runs on one instance (the others bale when the lock is held). The app sits in a permanent loop attempting to run the task every 2 seconds and the task itself takes 5 seconds, so I can see plenty of lock contention.
This works fine on my local computer so I've now packaged it up as an image and am attempting to run it on Google's Container Engine.
I'm configuring Hazelcast through Spring like so:
<hz:hazelcast id="instance">
<hz:config>
<hz:properties>
<hz:property name="hazelcast.discovery.enabled">true</hz:property>
</hz:properties>
<hz:network port="5701" port-auto-increment="true">
<hz:join>
<hz:multicast enabled="false" />
<hz:tcp-ip enabled="false" />
<hz:discovery-strategies>
<hz:discovery-strategy
class-name="com.noctarius.hazelcast.kubernetes.HazelcastKubernetesDiscoveryStrategy">
<hz:properties>
<hz:property name="service-dns">hazeltest-service-simple</hz:property>
<hz:property name="service-dns-timeout">10</hz:property>
</hz:properties>
</hz:discovery-strategy>
</hz:discovery-strategies>
</hz:join>
</hz:network>
</hz:config>
</hz:hazelcast>
The service definition I'm using looks like this:
apiVersion: v1
kind: Service
metadata:
labels:
name: hazeltest-name-label-simple
name: hazeltest-service-simple
spec:
ports:
- port: 5701
targetPort: 5701
protocol: TCP
selector:
app: hazeltest-app-label
When I run the app, I can see the following in the logs:
2017-02-08T11:37:42.435702737Z Feb 08, 2017 11:37:42 AM com.hazelcast.nio.tcp.SocketAcceptorThread
2017-02-08T11:37:42.435758276Z INFO: [10.0.0.17]:5701 [dev] [3.7.5] Accepting socket connection from /10.0.1.12:52664
2017-02-08T11:37:42.461609384Z Feb 08, 2017 11:37:42 AM com.hazelcast.nio.tcp.TcpIpConnectionManager
2017-02-08T11:37:42.461679256Z INFO: [10.0.0.17]:5701 [dev] [3.7.5] Established socket connection between /10.0.0.17:5701 and /10.0.1.12:52664
2017-02-08T11:37:42.462780759Z Feb 08, 2017 11:37:42 AM com.hazelcast.nio.tcp.TcpIpConnectionManager
2017-02-08T11:37:42.462796237Z WARNING: [10.0.0.17]:5701 [dev] [3.7.5] Wrong bind request from [10.0.1.12]:5701! This node is not requested endpoint: [10.3.250.127]:5701
2017-02-08T11:37:42.464388597Z Feb 08, 2017 11:37:42 AM com.hazelcast.nio.tcp.TcpIpConnection
2017-02-08T11:37:42.464411912Z INFO: [10.0.0.17]:5701 [dev] [3.7.5] Connection[id=8, /10.0.0.17:5701->/10.0.1.12:52664, endpoint=null, alive=false, type=MEMBER] closed. Reason: Wrong bind request from [10.0.1.12]:5701! This node is not requested endpoint: [10.3.250.127]:5701
I'm running Hazelcast 3.7.5 and version 0.9.3 of this plugin, if that helps. Any pointers on why this would be happening or how to resolve would be super-appreciated.
Is it working properly?
I think there should be more information about how this discovery plugin suppose to work inside Kubernetes. At the moment I can see the following logic:
- Plugin queries either DNS or REST for Service endpoint (I have used DNS)
- It gets Service IP + port
- Tries to establish connection with kubernetes service that randomly forwards query to backend hazelcast replicas in order to form the cluster
- ??? fails actually, because service itself is ephemeral and is not a member of the cluster
Here are logs from one of the running pods inside minikube where I have tried to combine it with management server:
$ kubectl logs hazelcast-server-d978ff479-vx4kk
Kubernetes Namespace: default
Kubernetes Service DNS:
'########################################
'# RUN_JAVA=
'# JAVA_OPTS=-Xms512M -Xmx2056M -Dcom.sun.management.jmxremote -'Dcom.sun.management.jmxremote.authenticate=false -Dhazelcast.diagnostics.enabled=true -'Dhazelcast.rest.enabled=true
'# CLASSPATH=/:/opt/hazelcast/:/opt/hazelcast/external/*:
'########################################
Checking custom configuration
no custom configuration found
Mar 06, 2018 6:47:52 AM com.hazelcast.config.XmlConfigLocator
INFO: Loading 'hazelcast.xml' from working directory.
Mar 06, 2018 6:47:52 AM com.hazelcast.instance.AddressPicker
INFO: [LOCAL] [dev] [3.9.3] Prefer IPv4 stack is true.
Mar 06, 2018 6:47:52 AM com.hazelcast.instance.AddressPicker
INFO: [LOCAL] [dev] [3.9.3] Picked [172.17.0.7]:5701, using socket ServerSocket[addr=/0.0.0.0,localport=5701], bind any local is true
Mar 06, 2018 6:47:53 AM com.hazelcast.system
INFO: [172.17.0.7]:5701 [dev] [3.9.3] Hazelcast 3.9.3 (20180216 - 539b124) starting at [172.17.0.7]:5701
Mar 06, 2018 6:47:53 AM com.hazelcast.system
INFO: [172.17.0.7]:5701 [dev] [3.9.3] Copyright (c) 2008-2018, Hazelcast, Inc. All Rights Reserved.
Mar 06, 2018 6:47:53 AM com.hazelcast.system
INFO: [172.17.0.7]:5701 [dev] [3.9.3] Configured Hazelcast Serialization version: 1
Mar 06, 2018 6:47:53 AM com.hazelcast.spi.impl.operationservice.impl.BackpressureRegulator
INFO: [172.17.0.7]:5701 [dev] [3.9.3] Backpressure is disabled
Mar 06, 2018 6:47:53 AM com.hazelcast.spi.discovery.integration.DiscoveryService
INFO: [172.17.0.7]:5701 [dev] [3.9.3] Kubernetes Discovery properties: { service-dns: hazelcast-service.default.svc.cluster.local, service-dns-timeout: 5, service-name: null, service-label: null, service-label-value: true, namespace: default, resolve-not-ready-addresses: false, kubernetes-master: https://kubernetes.default.svc}
Mar 06, 2018 6:47:53 AM com.hazelcast.spi.discovery.integration.DiscoveryService
INFO: [172.17.0.7]:5701 [dev] [3.9.3] Kubernetes Discovery activated resolver: DnsEndpointResolver
Mar 06, 2018 6:47:53 AM com.hazelcast.instance.Node
INFO: [172.17.0.7]:5701 [dev] [3.9.3] Activating Discovery SPI Joiner
Mar 06, 2018 6:47:54 AM com.hazelcast.spi.impl.operationexecutor.impl.OperationExecutorImpl
INFO: [172.17.0.7]:5701 [dev] [3.9.3] Starting 2 partition threads and 3 generic threads (1 dedicated for priority tasks)
Mar 06, 2018 6:47:54 AM com.hazelcast.internal.diagnostics.Diagnostics
INFO: [172.17.0.7]:5701 [dev] [3.9.3] Diagnostics started
Mar 06, 2018 6:47:54 AM com.hazelcast.internal.diagnostics.BuildInfoPlugin
INFO: [172.17.0.7]:5701 [dev] [3.9.3] Plugin:active
Mar 06, 2018 6:47:54 AM com.hazelcast.internal.diagnostics.SystemPropertiesPlugin
INFO: [172.17.0.7]:5701 [dev] [3.9.3] Plugin:active
Mar 06, 2018 6:47:54 AM com.hazelcast.internal.diagnostics.ConfigPropertiesPlugin
INFO: [172.17.0.7]:5701 [dev] [3.9.3] Plugin:active
Mar 06, 2018 6:47:54 AM com.hazelcast.internal.diagnostics.MetricsPlugin
INFO: [172.17.0.7]:5701 [dev] [3.9.3] Plugin:active, period-millis:60000
Mar 06, 2018 6:47:54 AM com.hazelcast.internal.diagnostics.SlowOperationPlugin
INFO: [172.17.0.7]:5701 [dev] [3.9.3] Plugin:active, period-millis:60000
Mar 06, 2018 6:47:54 AM com.hazelcast.internal.diagnostics.MemberHazelcastInstanceInfoPlugin
INFO: [172.17.0.7]:5701 [dev] [3.9.3] Plugin:active, period-millis:60000
Mar 06, 2018 6:47:54 AM com.hazelcast.internal.diagnostics.SystemLogPlugin
INFO: [172.17.0.7]:5701 [dev] [3.9.3] Plugin:active: logPartitions:false
Mar 06, 2018 6:47:54 AM com.hazelcast.internal.diagnostics.MemberHazelcastInstanceInfoPlugin
INFO: [172.17.0.7]:5701 [dev] [3.9.3] Plugin:active, period-millis:10000
Mar 06, 2018 6:47:54 AM com.hazelcast.internal.diagnostics.OperationHeartbeatPlugin
INFO: [172.17.0.7]:5701 [dev] [3.9.3] Plugin:active: period-millis:10000 max-deviation:33%
Mar 06, 2018 6:47:54 AM com.hazelcast.core.LifecycleService
INFO: [172.17.0.7]:5701 [dev] [3.9.3] [172.17.0.7]:5701 is STARTING
Mar 06, 2018 6:47:54 AM com.hazelcast.nio.tcp.TcpIpConnector
INFO: [172.17.0.7]:5701 [dev] [3.9.3] Connecting to /10.99.49.133:5701, timeout: 0, bind-any: true
Mar 06, 2018 6:47:55 AM com.hazelcast.nio.tcp.TcpIpConnectionManager
INFO: [172.17.0.7]:5701 [dev] [3.9.3] Established socket connection between /172.17.0.7:44347 and /10.99.49.133:5701
Mar 06, 2018 6:47:55 AM com.hazelcast.nio.tcp.TcpIpConnection
INFO: [172.17.0.7]:5701 [dev] [3.9.3] Connection[id=1, /172.17.0.7:44347->/10.99.49.133:5701, endpoint=[10.99.49.133]:5701, alive=false, type=MEMBER] closed. Reason: Connection closed by the other side
Mar 06, 2018 6:47:56 AM com.hazelcast.nio.tcp.TcpIpConnector
INFO: [172.17.0.7]:5701 [dev] [3.9.3] Connecting to /10.99.49.133:5701, timeout: 0, bind-any: true
Mar 06, 2018 6:47:57 AM com.hazelcast.nio.tcp.TcpIpConnectionManager
INFO: [172.17.0.7]:5701 [dev] [3.9.3] Established socket connection between /172.17.0.7:51935 and /10.99.49.133:5701
Mar 06, 2018 6:47:57 AM com.hazelcast.nio.tcp.TcpIpConnection
INFO: [172.17.0.7]:5701 [dev] [3.9.3] Connection[id=2, /172.17.0.7:51935->/10.99.49.133:5701, endpoint=[10.99.49.133]:5701, alive=false, type=MEMBER] closed. Reason: Connection closed by the other side
Mar 06, 2018 6:47:58 AM com.hazelcast.nio.tcp.TcpIpConnector
INFO: [172.17.0.7]:5701 [dev] [3.9.3] Connecting to /10.99.49.133:5701, timeout: 0, bind-any: true
Mar 06, 2018 6:47:59 AM com.hazelcast.system
INFO: [172.17.0.7]:5701 [dev] [3.9.3] Cluster version set to 3.9
Mar 06, 2018 6:47:59 AM com.hazelcast.internal.cluster.ClusterService
INFO: [172.17.0.7]:5701 [dev] [3.9.3]
Members {size:1, ver:1} [
Member [172.17.0.7]:5701 - 1cded4bf-6f7b-4bf8-92cc-c11645db87dc this
]Mar 06, 2018 6:47:59 AM com.hazelcast.internal.management.ManagementCenterService
INFO: [172.17.0.7]:5701 [dev] [3.9.3] Hazelcast will connect to Hazelcast Management Center on address:
http://localhost:8080/mancenter
Mar 06, 2018 6:47:59 AM com.hazelcast.core.LifecycleService
INFO: [172.17.0.7]:5701 [dev] [3.9.3] [172.17.0.7]:5701 is STARTED
Mar 06, 2018 6:47:59 AM com.hazelcast.internal.management.ManagementCenterService
INFO: [172.17.0.7]:5701 [dev] [3.9.3] Failed to pull tasks from management center
Mar 06, 2018 6:47:59 AM com.hazelcast.internal.management.ManagementCenterService
INFO: [172.17.0.7]:5701 [dev] [3.9.3] Failed to connect to:http://localhost:8080/mancenter/collector.do
Mar 06, 2018 6:48:01 AM com.hazelcast.nio.tcp.TcpIpConnectionManager
INFO: [172.17.0.7]:5701 [dev] [3.9.3] Established socket connection between /172.17.0.7:60675 and /10.99.49.133:5701
Mar 06, 2018 6:48:01 AM com.hazelcast.nio.tcp.TcpIpConnection
INFO: [172.17.0.7]:5701 [dev] [3.9.3] Connection[id=3, /172.17.0.7:60675->/10.99.49.133:5701, endpoint=[10.99.49.133]:5701, alive=false, type=MEMBER] closed. Reason: Connection closed by the other side
Q: Given scenario of changing replicas from 0 to 2+ ... members join/discover each other later than desired
Hi,
Thanks for the plugin, it is working really well.
One scenario I have noticed is that when there are no running pods with Hazelcast and we try to start with replicas of 2 or more (jump from 0 to 2+ replicas). In this scenario the initial discovery does not find any currently running pods to join and so the 2 replicas start without joining each other.
At some point later they join/discover each other. So it seems (and makes sense) that Hazelcast triggers/polls the discovery plugin periodically to find new nodes etc.
Now in our use case we are using a replicated map and we need to be careful to avoid this scenario (going from 0 pods to many pods) as the replicated maps are not in sync just at the point of pod startup and that is a problem for our use case. The replicated maps do come into sync at some point later (I believe Hazelcast polls the discovery plugin later and the nodes join and sync).
I wonder if others have looked at this and got other thoughts.
Cheers, Rob.
Port in DnsEndpointResolver is 0
I just used this module and deployed two services on google cloud, added a headless service which returns service records for both services but the hazelcast connect fails because the DnsEndpointResolver tries to connect to port 0 - the service itself exposes port 5701.
Any thoughts?
Hazelcast port not configurable
When using the DnsEndpointResolver, the Hazelcast port cannot be configured: the DnsEndpointResolver.getHazelcastPort() method always returns the static DEFAULT_PORT value. Could this port value be configurable, just as it is when using the ServiceEndpointResolver?
Please sign the CLA (Contributors License Agreement)
Hey guys,
We would like to go forward and officially support the kubernetes plugin. That includes moving the plugin repository to the Hazelcast GitHub account and handing over responsibility to our integrations team.
To make this step it would be great if you could help out and sign our Contributors License Agreement to give us full permissions over the contributions you did. I really appreciate your help and time and hope to see more contributions in the future :)
CLA necessary from:
- @jimmidyson *
- @amoAHCP *
- @ReToCode *
- @yurybubnov
- @mrahbar (since I would like to merge your PR as well) *
You can find information as well as the instructions on our public part of the wiki: https://hazelcast.atlassian.net/wiki/display/COM/Hazelcast+Contributor+Agreement
Thanks again and I hope to see further PRs coming along :)
* already signed
Hazelcast starts in Standalone mode in 3.9.2
In the example XML you have
<!-- only necessary prior Hazelcast 3.8 -->
<property name="hazelcast.discovery.enabled">true</property>
This is misleading, we needed to have the property in 3.9 as well. Without it, we'd get a log message saying "No join methods enabled, starting in standalone mode." and the cluster would fail to initialize.
Switch to SRV lookups for service endpoints
Currently connections will be made to the service IP rather than the pod IPs. Switching to SRV records will enable discovery of pod IPs & ports from DNS.
I'll submit a PR to see how this looks & what you think.
Integration with kubernetes horizontal pod autoscaling
This is a feature request to integrate Hazelcast with Kubernete's horizontal pod autoscaling (HPA).
The newer autoscaling/v2beta1 version of HPA allows for memory and custom metrics to be used instead of CPU utilization, however, some experiments with this API illustrated that HZ pods could scale up but did not scale down (i.e. they did not trigger a scale down threshold).
This feature request is for an investigation, design and implementation on how to best integrate HZ's metrics with kubernetes such that both scaling up and down occurs.
Include Hazelcast Kubernetes in hazelcast-all
NPE received during Hazelcast start up
During initialization of Hazelcast server we got the following exception:
java.lang.NullPointerException at com.noctarius.hazelcast.kubernetes.ServiceEndpointResolver.getDiscoveryNodes(ServiceEndpointResolver.java:95 at com.noctarius.hazelcast.kubernetes.ServiceEndpointResolver.resolve(ServiceEndpointResolver.java:75 at com.noctarius.hazelcast.kubernetes.HazelcastKubernetesDiscoveryStrategy.discoverNodes(HazelcastKubernetesDiscoveryStrategy.java:88 at com.hazelcast.spi.discovery.impl.DefaultDiscoveryService.discoverNodes(DefaultDiscoveryService.java:74 at com.hazelcast.client.spi.impl.discovery.DiscoveryAddressProvider.loadAddresses(DiscoveryAddressProvider.java:44 at com.hazelcast.client.spi.impl.ClusterListenerSupport.getSocketAddresses(ClusterListenerSupport.java:174 at com.hazelcast.client.spi.impl.ClusterListenerSupport.connect(ClusterListenerSupport.java:233 at com.hazelcast.client.spi.impl.ClusterListenerSupport.connectToOne(ClusterListenerSupport.java:209 at com.hazelcast.client.spi.impl.ClusterListenerSupport.connectToCluster(ClusterListenerSupport.java:161 at com.hazelcast.client.spi.impl.ClientClusterServiceImpl.start(ClientClusterServiceImpl.java:203 at com.hazelcast.client.impl.HazelcastClientInstanceImpl.start(HazelcastClientInstanceImpl.java:303 at com.idea2.distributed.DistributedServices.restart(DistributedServices.java:311 at com.idea2.distributed.exceptions.ExceptionsUtils.rethrow(ExceptionsUtils.java:38 at com.idea2.distributed.exceptions.ExceptionsUtils.manageError(ExceptionsUtils.java:26 at com.idea2.distributed.Queues.getQueue(Queues.java:24 at com.idea2.platformmanager.logic.tasks.framework.TasksListener.lambda$0(TasksListener.java:45 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617 at java.lang.Thread.run(Thread.java:745) Context
The service config that it is used for discover is:
Name: hazelcast Namespace: default Labels: <none> Selector: type=hazelcast Type: ClusterIP IP: 10.239.255.186 Port: hazel 5701/TCP Endpoints: <none> Session Affinity: None No events.
fix document about the dns lookup to use headless service
As I test, this plugin only works with headless service name when using dns lookup. Please fix the document.
Thanks
Project Dependecies
Hi,
I have been using your plugin with great success. 👍
I had less luck in configuring the plugin to use the kubernetes api due to RBAC user permission on the service. However the DNS lookup worked flawlessly and i don't need to anything.
I'm using this version (gradle)
compile group: 'com.hazelcast', name: 'hazelcast-kubernetes', version: '1.1.0'
When gradle resolved the dependecies it also pulls in this:
//https://mvnrepository.com/artifact/com.hazelcast/hazelcast-kubernetes/1.1.0
io.fabric8:kubernetes-client:3.0.3
Looking at the project, this is not used anymore.
Would it be possible to get a new build with this dependency removed?
Regards
ade
Typo in sample code: incorrect class name
I just want to report a typo on the main Github page of this plugin that caused me some headaches getting this plugin working.
In the DNS Lookup section of README.adoc, the discovery-strategy class name is incorrectly set to com.hazelcast.HazelcastKubernetesDiscoveryStrategy, where it should be set to com.hazelcast.kubernetes.HazelcastKubernetesDiscoveryStrategy instead.
Hazelcast doesn't throw any Exceptions or logs any errors whatsoever, which makes this error extremely annoying to realize.
Established socket connection
Why my cluster nodes cannot connect to each other? Every hazelcast instances have the same group name.
Is there a difference for hazelcast and discovery plugin between which kubernetes network do I use?
My configs:
@Bean
public Config hazelcastConfig() {
Config config = new Config();
config.getGroupConfig()
.setName("mobiry-cluster")
.setPassword("password");
config.setProperty("hazelcast.discovery.enabled", "true");
JoinConfig joinConfig = config.getNetworkConfig().getJoin();
joinConfig.getMulticastConfig().setEnabled(false);
joinConfig.getTcpIpConfig().setEnabled(false);
DiscoveryConfig discoveryConfig = joinConfig.getDiscoveryConfig();
HazelcastKubernetesDiscoveryStrategyFactory factory = new HazelcastKubernetesDiscoveryStrategyFactory();
DiscoveryStrategyConfig strategyConfig = new DiscoveryStrategyConfig(factory);
strategyConfig.addProperty("service-name", "mobiry");
strategyConfig.addProperty("service-label-name", "run");
strategyConfig.addProperty("service-label-value", "mobiry");
strategyConfig.addProperty("namespace", "default");
discoveryConfig.addDiscoveryStrategyConfig(strategyConfig);
return config;
}
@Bean
public WebFilter webFilter(HazelcastInstance hazelcastInstance) {
Properties properties = new Properties();
properties.put("instance-name", hazelcastInstance.getName());
properties.put("sticky-session", "false");
return new WebFilter(properties);
}
Logs:
Members [1] {
Member [100.97.75.211]:5701 - 6d545348-b240-40a8-92ed-9214ae00287e this
}
16-Aug-2018 12:04:41.200 INFO [main] com.hazelcast.core.LifecycleService.null [100.97.75.211]:5701 [dev] [3.8.7] [100.97.75.211]:5701 is STARTED
16-Aug-2018 12:04:41.200 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service [Catalina]
16-Aug-2018 12:04:41.200 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet Engine: Apache Tomcat/8.5.32
16-Aug-2018 12:04:41.239 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application archive [/usr/local/tomcat/webapps/ROOT.war]
16-Aug-2018 12:04:42.990 INFO [hz.SESSION-REPLICATION-INSTANCE.IO.thread-Acceptor] com.hazelcast.nio.tcp.SocketAcceptorThread.null [100.97.75.211]:5701 [dev] [3.8.7] Accepting socket connection from /100.112.0.83:38101
16-Aug-2018 12:04:42.999 INFO [hz.SESSION-REPLICATION-INSTANCE.cached.thread-3] com.hazelcast.nio.tcp.TcpIpConnectionManager.null [100.97.75.211]:5701 [dev] [3.8.7] Established socket connection between /100.97.75.211:5701 and /100.112.0.83:38101
16-Aug-2018 12:04:44.515 INFO [localhost-startStop-1] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
16-Aug-2018 12:04:44.524 INFO [localhost-startStop-1] org.apache.catalina.core.ApplicationContext.log No Spring WebApplicationInitializer types detected on classpath
16-Aug-2018 12:04:44.541 INFO [localhost-startStop-1] com.hazelcast.session.HazelcastSessionManager.startInternal contextPath:
16-Aug-2018 12:04:44.557 INFO [localhost-startStop-1] com.hazelcast.session.HazelcastSessionManager.startInternal HazelcastSessionManager started...
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
16-Aug-2018 12:04:44.597 INFO [localhost-startStop-1] org.apache.catalina.core.ApplicationContext.log Initializing Spring FrameworkServlet 'spring-web'
16-Aug-2018 12:04:45.953 INFO [localhost-startStop-1] com.hazelcast.instance.AddressPicker.null [LOCAL] [mobiry-cluster] [3.8.7] Prefer IPv4 stack is true.
16-Aug-2018 12:04:45.957 INFO [localhost-startStop-1] com.hazelcast.instance.AddressPicker.null [LOCAL] [mobiry-cluster] [3.8.7] Picked [100.97.75.211]:5702, using socket ServerSocket[addr=/0:0:0:0:0:0:0:0,localport=5702], bind any local is true
16-Aug-2018 12:04:45.985 INFO [localhost-startStop-1] com.hazelcast.system.null [100.97.75.211]:5702 [mobiry-cluster] [3.8.7] Hazelcast 3.8.7 (20171109 - 0fe6297) starting at [100.97.75.211]:5702
16-Aug-2018 12:04:45.985 INFO [localhost-startStop-1] com.hazelcast.system.null [100.97.75.211]:5702 [mobiry-cluster] [3.8.7] Copyright (c) 2008-2016, Hazelcast, Inc. All Rights Reserved.
16-Aug-2018 12:04:45.985 INFO [localhost-startStop-1] com.hazelcast.system.null [100.97.75.211]:5702 [mobiry-cluster] [3.8.7] Configured Hazelcast Serialization version : 1
16-Aug-2018 12:04:46.202 INFO [localhost-startStop-1] com.hazelcast.spi.impl.operationservice.impl.BackpressureRegulator.null [100.97.75.211]:5702 [mobiry-cluster] [3.8.7] Backpressure is disabled
16-Aug-2018 12:04:46.658 INFO [localhost-startStop-1] com.hazelcast.spi.discovery.integration.DiscoveryService.null [100.97.75.211]:5702 [mobiry-cluster] [3.8.7] Kubernetes Discovery properties: { service-dns: null, service-dns-timeout: 5, service-name: mobiry, service-label: run, service-label-value: mobiry, namespace: default, resolve-not-ready-addresses: false, kubernetes-master: https://kubernetes.default.svc}
16-Aug-2018 12:04:46.659 INFO [localhost-startStop-1] com.hazelcast.spi.discovery.integration.DiscoveryService.null [100.97.75.211]:5702 [mobiry-cluster] [3.8.7] Kubernetes Discovery: Bearer Token { eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImRlZmF1bHQtdG9rZW4teng0aGQiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGVmYXVsdCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6ImUxNzE2ZWM5LWEwOTItMTFlOC05ZGFmLTEyOGZjY2FkYWRiOCIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0OmRlZmF1bHQifQ.2OcyUxuBYHozVSOMrq-ghquLD42nR651Hic_Yqi_UlcpzifryN0O1i3Iz3JBU8MdEanI0lP2olF1HR5k0_DBbqFcX-k1BaiK3AfJtArBd7yn0fDHvJ763MQ14TPeYFNQIk0Eihghh_EhJPHYLhaVCwGz5HSbahvPpf_JvV6uPV6fFRS9mZCUxJCyNj4DV09Xg9Y8i2ZNDncDnhQrvqTKjM0pz3Hwas3ywAs924jHNAn6HXJNxwwD0JjTr_UUFsoYslIW6MtMzOzUdrdtRGx-y0vq7Eu9ACywRgSsxzeNwNt0XqxP6KrbzQQ3-EMQdDIJzBbld_d0pNiG1tLXD7usBQ }
16-Aug-2018 12:04:46.957 INFO [localhost-startStop-1] com.hazelcast.spi.discovery.integration.DiscoveryService.null [100.97.75.211]:5702 [mobiry-cluster] [3.8.7] Kubernetes Discovery activated resolver: ServiceEndpointResolver
16-Aug-2018 12:04:46.958 INFO [localhost-startStop-1] com.hazelcast.spi.discovery.integration.DiscoveryService.null [100.97.75.211]:5702 [mobiry-cluster] [3.8.7] Kubernetes Discovery properties: { service-dns: null, service-dns-timeout: 5, service-name: mobiry, service-label: run, service-label-value: mobiry, namespace: default, resolve-not-ready-addresses: false, kubernetes-master: https://kubernetes.default.svc}
16-Aug-2018 12:04:46.958 INFO [localhost-startStop-1] com.hazelcast.spi.discovery.integration.DiscoveryService.null [100.97.75.211]:5702 [mobiry-cluster] [3.8.7] Kubernetes Discovery: Bearer Token { eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImRlZmF1bHQtdG9rZW4teng0aGQiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGVmYXVsdCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6ImUxNzE2ZWM5LWEwOTItMTFlOC05ZGFmLTEyOGZjY2FkYWRiOCIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0OmRlZmF1bHQifQ.2OcyUxuBYHozVSOMrq-ghquLD42nR651Hic_Yqi_UlcpzifryN0O1i3Iz3JBU8MdEanI0lP2olF1HR5k0_DBbqFcX-k1BaiK3AfJtArBd7yn0fDHvJ763MQ14TPeYFNQIk0Eihghh_EhJPHYLhaVCwGz5HSbahvPpf_JvV6uPV6fFRS9mZCUxJCyNj4DV09Xg9Y8i2ZNDncDnhQrvqTKjM0pz3Hwas3ywAs924jHNAn6HXJNxwwD0JjTr_UUFsoYslIW6MtMzOzUdrdtRGx-y0vq7Eu9ACywRgSsxzeNwNt0XqxP6KrbzQQ3-EMQdDIJzBbld_d0pNiG1tLXD7usBQ }
16-Aug-2018 12:04:46.976 INFO [localhost-startStop-1] com.hazelcast.spi.discovery.integration.DiscoveryService.null [100.97.75.211]:5702 [mobiry-cluster] [3.8.7] Kubernetes Discovery activated resolver: ServiceEndpointResolver
16-Aug-2018 12:04:46.976 INFO [localhost-startStop-1] com.hazelcast.instance.Node.null [100.97.75.211]:5702 [mobiry-cluster] [3.8.7] Activating Discovery SPI Joiner
16-Aug-2018 12:04:47.146 INFO [localhost-startStop-1] com.hazelcast.spi.impl.operationexecutor.impl.OperationExecutorImpl.null [100.97.75.211]:5702 [mobiry-cluster] [3.8.7] Starting 2 partition threads
16-Aug-2018 12:04:47.162 INFO [localhost-startStop-1] com.hazelcast.spi.impl.operationexecutor.impl.OperationExecutorImpl.null [100.97.75.211]:5702 [mobiry-cluster] [3.8.7] Starting 3 generic threads (1 dedicated for priority tasks)
16-Aug-2018 12:04:47.176 INFO [localhost-startStop-1] com.hazelcast.core.LifecycleService.null [100.97.75.211]:5702 [mobiry-cluster] [3.8.7] [100.97.75.211]:5702 is STARTING
16-Aug-2018 12:04:47.610 INFO [hz._hzInstance_1_mobiry-cluster.cached.thread-3] com.hazelcast.nio.tcp.InitConnectionTask.null [100.97.75.211]:5702 [mobiry-cluster] [3.8.7] Connecting to /100.97.75.211:5701, timeout: 0, bind-any: true
16-Aug-2018 12:04:47.611 INFO [hz.SESSION-REPLICATION-INSTANCE.IO.thread-Acceptor] com.hazelcast.nio.tcp.SocketAcceptorThread.null [100.97.75.211]:5701 [dev] [3.8.7] Accepting socket connection from /100.97.75.211:58316
16-Aug-2018 12:04:47.612 INFO [hz.SESSION-REPLICATION-INSTANCE.cached.thread-2] com.hazelcast.nio.tcp.TcpIpConnectionManager.null [100.97.75.211]:5701 [dev] [3.8.7] Established socket connection between /100.97.75.211:5701 and /100.97.75.211:58316
16-Aug-2018 12:04:47.622 INFO [hz._hzInstance_1_mobiry-cluster.cached.thread-3] com.hazelcast.nio.tcp.TcpIpConnectionManager.null [100.97.75.211]:5702 [mobiry-cluster] [3.8.7] Established socket connection between /100.97.75.211:58316 and /100.97.75.211:5701
16-Aug-2018 12:04:47.624 INFO [hz._hzInstance_1_mobiry-cluster.cached.thread-2] com.hazelcast.nio.tcp.InitConnectionTask.null [100.97.75.211]:5702 [mobiry-cluster] [3.8.7] Connecting to /100.112.0.83:5701, timeout: 0, bind-any: true
16-Aug-2018 12:04:47.625 INFO [hz._hzInstance_1_mobiry-cluster.cached.thread-2] com.hazelcast.nio.tcp.TcpIpConnectionManager.null [100.97.75.211]:5702 [mobiry-cluster] [3.8.7] Established socket connection between /100.97.75.211:49449 and /100.112.0.83:5701
16-Aug-2018 12:04:48.632 WARNING [hz.SESSION-REPLICATION-INSTANCE.priority-generic-operation.thread-0] com.hazelcast.internal.cluster.impl.ClusterJoinManager.null [100.97.75.211]:5701 [dev] [3.8.7] Received an invalid join request from [100.97.75.211]:5702, cause: clusters part of different cluster-groups
16-Aug-2018 12:04:48.634 WARNING [hz.SESSION-REPLICATION-INSTANCE.priority-generic-operation.thread-0] com.hazelcast.internal.cluster.impl.ClusterJoinManager.null [100.97.75.211]:5701 [dev] [3.8.7] Received an invalid join request from [100.97.75.211]:5702, cause: clusters part of different cluster-groups
16-Aug-2018 12:04:48.636 INFO [hz._hzInstance_1_mobiry-cluster.generic-operation.thread-1] com.hazelcast.nio.tcp.TcpIpConnection.null [100.97.75.211]:5702 [mobiry-cluster] [3.8.7] Connection[id=1, /100.97.75.211:58316->/100.97.75.211:5701, endpoint=[100.97.75.211]:5701, alive=false, type=MEMBER] closed. Reason: Node could not join cluster at node: [100.97.75.211]:5701 Cause: the target cluster has a different group-name
16-Aug-2018 12:04:48.639 INFO [hz._hzInstance_1_mobiry-cluster.generic-operation.thread-0] com.hazelcast.nio.tcp.TcpIpConnection.null [100.97.75.211]:5702 [mobiry-cluster] [3.8.7] Connection[id=2, /100.97.75.211:49449->/100.112.0.83:5701, endpoint=[100.112.0.83]:5701, alive=false, type=MEMBER] closed. Reason: Node could not join cluster at node: [100.112.0.83]:5701 Cause: the target cluster has a different group-name
16-Aug-2018 12:04:48.639 WARNING [hz._hzInstance_1_mobiry-cluster.priority-generic-operation.thread-0] com.hazelcast.cluster.null [100.97.75.211]:5702 [mobiry-cluster] [3.8.7] Node could not join cluster at node: [100.97.75.211]:5701 Cause: the target cluster has a different group-name
16-Aug-2018 12:04:48.640 INFO [hz._hzInstance_1_mobiry-cluster.priority-generic-operation.thread-0] com.hazelcast.internal.cluster.impl.DiscoveryJoiner.null [100.97.75.211]:5702 [mobiry-cluster] [3.8.7] [100.97.75.211]:5701 is added to the blacklist.
16-Aug-2018 12:04:48.640 WARNING [hz._hzInstance_1_mobiry-cluster.priority-generic-operation.thread-0] com.hazelcast.cluster.null [100.97.75.211]:5702 [mobiry-cluster] [3.8.7] Node could not join cluster at node: [100.112.0.83]:5701 Cause: the target cluster has a different group-name
16-Aug-2018 12:04:48.640 INFO [hz._hzInstance_1_mobiry-cluster.priority-generic-operation.thread-0] com.hazelcast.internal.cluster.impl.DiscoveryJoiner.null [100.97.75.211]:5702 [mobiry-cluster] [3.8.7] [100.112.0.83]:5701 is added to the blacklist.
16-Aug-2018 12:04:48.640 INFO [localhost-startStop-1] com.hazelcast.system.null [100.97.75.211]:5702 [mobiry-cluster] [3.8.7] Cluster version set to 3.8
16-Aug-2018 12:04:48.641 INFO [hz.SESSION-REPLICATION-INSTANCE.IO.thread-in-1] com.hazelcast.nio.tcp.TcpIpConnection.null [100.97.75.211]:5701 [dev] [3.8.7] Connection[id=2, /100.97.75.211:5701->/100.97.75.211:58316, endpoint=[100.97.75.211]:5702, alive=false, type=MEMBER] closed. Reason: Connection closed by the other side
16-Aug-2018 12:04:48.642 INFO [localhost-startStop-1] com.hazelcast.internal.cluster.impl.DiscoveryJoiner.null [100.97.75.211]:5702 [mobiry-cluster] [3.8.7]
Members [1] {
Member [100.97.75.211]:5702 - 4c9e1694-6ba5-4079-a9c0-741a08040a0f this
}
16-Aug-2018 12:04:48.644 WARNING [hz._hzInstance_1_mobiry-cluster.generic-operation.thread-1] com.hazelcast.cluster.null [100.97.75.211]:5702 [mobiry-cluster] [3.8.7] Node could not join cluster at node: [100.97.75.211]:5701 Cause: the target cluster has a different group-name
16-Aug-2018 12:04:48.644 INFO [hz._hzInstance_1_mobiry-cluster.generic-operation.thread-1] com.hazelcast.internal.cluster.impl.DiscoveryJoiner.null [100.97.75.211]:5702 [mobiry-cluster] [3.8.7] [100.97.75.211]:5701 is added to the blacklist.
16-Aug-2018 12:04:48.643 WARNING [hz._hzInstance_1_mobiry-cluster.generic-operation.thread-0] com.hazelcast.cluster.null [100.97.75.211]:5702 [mobiry-cluster] [3.8.7] Node could not join cluster at node: [100.112.0.83]:5701 Cause: the target cluster has a different group-name
16-Aug-2018 12:04:48.644 INFO [hz._hzInstance_1_mobiry-cluster.generic-operation.thread-0] com.hazelcast.internal.cluster.impl.DiscoveryJoiner.null [100.97.75.211]:5702 [mobiry-cluster] [3.8.7] [100.112.0.83]:5701 is added to the blacklist.
16-Aug-2018 12:04:48.644 WARNING [localhost-startStop-1] com.hazelcast.instance.Node.null [100.97.75.211]:5702 [mobiry-cluster] [3.8.7] Config seed port is 5701 and cluster size is 1. Some of the ports seem occupied!
16-Aug-2018 12:04:48.680 INFO [localhost-startStop-1] com.hazelcast.core.LifecycleService.null [100.97.75.211]:5702 [mobiry-cluster] [3.8.7] [100.97.75.211]:5702 is STARTED
16-Aug-2018 12:04:48.779 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deployment of web application archive [/usr/local/tomcat/webapps/ROOT.war] has finished in [7,539] ms
16-Aug-2018 12:04:48.782 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-80"]
16-Aug-2018 12:04:48.789 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["ajp-nio-8009"]
16-Aug-2018 12:04:48.791 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in 11191 ms
16-Aug-2018 12:09:47.142 INFO [hz._hzInstance_1_mobiry-cluster.cached.thread-1] com.hazelcast.nio.tcp.TcpIpConnectionManager.null [100.97.75.211]:5702 [mobiry-cluster] [3.8.7] Established socket connection between /100.97.75.211:56246 and /100.97.75.211:5701
16-Aug-2018 12:09:47.142 INFO [hz.SESSION-REPLICATION-INSTANCE.IO.thread-Acceptor] com.hazelcast.nio.tcp.SocketAcceptorThread.null [100.97.75.211]:5701 [dev] [3.8.7] Accepting socket connection from /100.97.75.211:56246
16-Aug-2018 12:09:47.142 INFO [hz.SESSION-REPLICATION-INSTANCE.cached.thread-3] com.hazelcast.nio.tcp.TcpIpConnectionManager.null [100.97.75.211]:5701 [dev] [3.8.7] Established socket connection between /100.97.75.211:5701 and /100.97.75.211:56246
16-Aug-2018 12:09:47.164 INFO [hz._hzInstance_1_mobiry-cluster.cached.thread-1] com.hazelcast.nio.tcp.TcpIpConnectionManager.null [100.97.75.211]:5702 [mobiry-cluster] [3.8.7] Established socket connection between /100.97.75.211:43313 and /100.112.0.83:5701
NPE when no ready pods reported in Endpoints
Note
Potentially duplicate of #29
Scenario
When an HZ instance boots up and the ServiceEndpointResolver kicks in with a provided serviceName the retrieved Endpoints object via fabric8 Kubernetes-Client is iterated over endpoints.getSubsets() and endpointSubset.getAddresses(). But the latter is null because there are no ready endpoints. Raw json response:
{
"kind": "Endpoints",
"apiVersion": "v1",
"metadata": {
"name": "demo-service-9aadd30",
"namespace": "dev",
"selfLink": "/api/v1/namespaces/dev/endpoints/demo-service-9aadd30",
"uid": "8a4fcf99-92ea-11e7-846d-005056b85198",
"resourceVersion": "10935908",
"creationTimestamp": "2017-09-06T10:02:49Z",
"labels": {
"appName": "demo-service",
"chart": "demo-service-1.0.7-SNAPSHOT",
"release": "demo-service-9aadd30"
}
},
"subsets": [
{
"notReadyAddresses": [
{
"ip": "172.16.1.60",
"nodeName": "node03",
"targetRef": {
"kind": "Pod",
"namespace": "dev",
"name": "demo-service-9aadd30-3817585861-dd1zz",
"uid": "4abc9584-92f5-11e7-846d-005056b85198",
"resourceVersion": "10935516"
}
},
{
"ip": "172.16.1.61",
"nodeName": "comp03",
"targetRef": {
"kind": "Pod",
"namespace": "dev",
"name": "demo-service-9aadd30-test",
"uid": "c4f0c539-9302-11e7-bbe2-005056b89983",
"resourceVersion": "10935907"
}
},
{
"ip": "172.16.3.56",
"nodeName": "comp05",
"targetRef": {
"kind": "Pod",
"namespace": "dev",
"name": "demo-service-9aadd30-3817585861-w56dd",
"uid": "4ab4c940-92f5-11e7-846d-005056b85198",
"resourceVersion": "10935514"
}
},
{
"ip": "172.16.8.53",
"nodeName": "comp06",
"targetRef": {
"kind": "Pod",
"namespace": "dev",
"name": "demo-service-9aadd30-2520348926-b9d5p",
"uid": "d0ef3797-92f4-11e7-846d-005056b85198",
"resourceVersion": "10935702"
}
}
],
"ports": [
{
"name": "service-port",
"port": 8080,
"protocol": "TCP"
},
{
"name": "hazelcast-port",
"port": 5701,
"protocol": "TCP"
}
]
}
]
}
The same request but with a fake bootstrap-pod with matching selectors yields:
{
"kind": "Endpoints",
"apiVersion": "v1",
"metadata": {
"name": "demo-service-9aadd30",
"namespace": "dev",
"selfLink": "/api/v1/namespaces/dev/endpoints/demo-service-9aadd30",
"uid": "8a4fcf99-92ea-11e7-846d-005056b85198",
"resourceVersion": "10936092",
"creationTimestamp": "2017-09-06T10:02:49Z",
"labels": {
"appName": "demo-service",
"chart": "demo-service-1.0.7-SNAPSHOT",
"release": "demo-service-9aadd30"
}
},
"subsets": [
{
"addresses": [
{
"ip": "172.16.1.61",
"nodeName": "comp03",
"targetRef": {
"kind": "Pod",
"namespace": "dev",
"name": "demo-service-9aadd30-test",
"uid": "c4f0c539-9302-11e7-bbe2-005056b89983",
"resourceVersion": "10936091"
}
}
],
"notReadyAddresses": [
{
"ip": "172.16.1.60",
"nodeName": "comp03",
"targetRef": {
"kind": "Pod",
"namespace": "dev",
"name": "demo-service-9aadd30-3817585861-dd1zz",
"uid": "4abc9584-92f5-11e7-846d-005056b85198",
"resourceVersion": "10935516"
}
},
{
"ip": "172.16.3.56",
"nodeName": "comp05",
"targetRef": {
"kind": "Pod",
"namespace": "dev",
"name": "demo-service-9aadd30-3817585861-w56dd",
"uid": "4ab4c940-92f5-11e7-846d-005056b85198",
"resourceVersion": "10935514"
}
},
{
"ip": "172.16.8.53",
"nodeName": "comp06",
"targetRef": {
"kind": "Pod",
"namespace": "dev",
"name": "demo-service-9aadd30-2520348926-b9d5p",
"uid": "d0ef3797-92f4-11e7-846d-005056b85198",
"resourceVersion": "10935702"
}
}
],
"ports": [
{
"name": "service-port",
"port": 8080,
"protocol": "TCP"
},
{
"name": "hazelcast-port",
"port": 5701,
"protocol": "TCP"
}
]
}
]
}
And the deployment is successfull.
Culprint
Internal fabric8 uses Jackson to unmarshel json to Endpoints an non-existing addresses JSON-objects results to a null value for the addresses field. Hence
Possible solution
Add a null check on each subsets loop. e.g. :
if (endpointSubset.getAddresses() == null) {
continue;
}
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.