hazelcast / hazelcast-kubernetes Goto Github PK

View Code? Open in Web Editor NEW

172.0 49.0 99.0 520 KB

Kubernetes Discovery for Hazelcast

License: Apache License 2.0

Java 100.00%

kubernetes hazelcast cloud-native microservices java

hazelcast-kubernetes's People

Stargazers

Watchers

Forkers

kobalski corentin59 atomfrede amoahcp yurybubnov mrahbar fiksn chrba oscerd matf iteeconsulting emrahkocaman codepfleger barnyrelph emre-aydin st0ne-dot-at amiranda-ob asciijungle yohanesws ygxue lburgazzoli e-medicus isurulucky zfl926 xqianwang googlielmo marwestermann gamoshi-archives avalcepina pratyushamenaka mistic92 tstrilka renatoapcosta chaldeos leszko mmuruganandam topikachu mesutcelik tkgreg idanbauer newbare william-tran nmarasoiu ade90036 flyhard bardo1 eminn kwart sumhua mauna-ai zxshinxz ptzagk klubii henrikarro mustafaiman gasgithub metaver5o holmistr yeshm cut-to-the-cheese saindavs geertvanheusden adhesivee xuxinran rzs840707 azatkhakimov enozcan diegoep kennymacleod hepyu mouli-sanketh-maturi rafalleszko sumnerib purplewe jkremser timmhirsens doytsujin alparslanavci rajivbansal2981 janvanryswyckcegeka npomaroli zpqrtbnk arjen-iw gefgitacc1 sudarsan-sridharan alexrogalskiy payara fawazghali sadilchamishka xupengtony isabella232 kambaa linyanshi nagu002 ismart-yuxi anonymous-coward

hazelcast-kubernetes's Issues

Enable external Smart Client (expose PODs and assign public IP automatically using Kubernetes API)

Currently, if someone deploys a Hazelcast cluster (manually or via Helm Chart), it's accessible via LoadBalancer service and therefore available only with the use of Dummy Client. That is the standard way how Kubernetes deals with multiple PODs. They are accessible only via one service [1].

It should be possible to expose each POD separately (using the Kubernetes Adds-On "Service Per POD" [2]). Then, we could use Kubernetes API to get the information about external IPs and set it in discoverNodes() or AddressPicker implementation.

[1] https://stackoverflow.com/questions/45650526/kubernetes-expose-only-single-pod-of-statefulset
[2] https://github.com/GoogleCloudPlatform/metacontroller/tree/master/examples/service-per-pod

Performance degradation seen using plugin

I see a performance degradation using plugin comparing to use a standalone mode of Hazelcast. Degradation is about 2x-3x slower.

My server side config:

Config config = com.hazelcast.config.Config();
config.setProperty("hazelcast.discovery.enabled", "true");
NetworkConfig networkConfig = config.getNetworkConfig();
networkConfig.getJoin().getMulticastConfig().setEnabled(false);
networkConfig.getJoin().getTcpIpConfig().setEnabled(false);
DiscoveryStrategyConfig discoveryStrategyConfig = new DiscoveryStrategyConfig(new HazelcastKubernetesDiscoveryStrategyFactory());
String serviceName = "hazelcast";
discoveryStrategyConfig.addProperty("service-name", serviceName);
DiscoveryConfig discoveryConfig = new DiscoveryConfig();
discoveryConfig.addDiscoveryStrategyConfig(discoveryStrategyConfig);
networkConfig.getJoin().setDiscoveryConfig(discoveryConfig);

My client side config:

ClientConfig clientConfig = new ClientConfig();
ClientNetworkConfig networkConfigCloud = new ClientNetworkConfig();
clientConfig.setProperty("hazelcast.discovery.enabled", "true");
DiscoveryStrategyConfig discoveryStrategyConfig = new DiscoveryStrategyConfig(new HazelcastKubernetesDiscoveryStrategyFactory());
String serviceName = "hazelcast";
discoveryStrategyConfig.addProperty("service-name", serviceName);
DiscoveryConfig discoveryConfig = new DiscoveryConfig();
discoveryConfig.addDiscoveryStrategyConfig(discoveryStrategyConfig);
networkConfigCloud.setDiscoveryConfig(discoveryConfig);

Some parameters:

Service name called "hazelcast" is a Kubernetes service.
Using Hazelcast 3.6.
Using Kubernetes 1.5.1.

Am I missing something? What could be the issue?

Thanks in advance.

Implement retry pattern for REST API calls

Sometimes we get the following exception under an heavy load context:

Error executing: GET at: https://kubernetes.default.svc/api/v1/namespaces/default/endpoints/hazelcast. Cause: Failed to connect to kubernetes.default.svc/10.239.240.1:443 , io.fabric8.kubernetes.client.KubernetesClientException: Error executing: GET at: https://kubernetes.default.svc/api/v1/namespaces/default/endpoints/hazelcast. Cause: Failed to connect to kubernetes.default.svc/10.239.240.1:443 at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestException(OperationSupport.java:272
 at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:205
 at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:196
 at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleGet(BaseOperation.java:483
 at io.fabric8.kubernetes.client.dsl.base.BaseOperation.get(BaseOperation.java:108
 at com.noctarius.hazelcast.kubernetes.ServiceEndpointResolver.resolve(ServiceEndpointResolver.java:71
 at com.noctarius.hazelcast.kubernetes.HazelcastKubernetesDiscoveryStrategy.discoverNodes(HazelcastKubernetesDiscoveryStrategy.java:88
 at com.hazelcast.spi.discovery.impl.DefaultDiscoveryService.discoverNodes(DefaultDiscoveryService.java:74
 at com.hazelcast.client.spi.impl.discovery.DiscoveryAddressTranslator.refresh(DiscoveryAddressTranslator.java:67
 at com.hazelcast.client.spi.impl.discovery.DiscoveryAddressTranslator.translate(DiscoveryAddressTranslator.java:56
 at com.hazelcast.client.connection.nio.ClientConnectionManagerImpl.getOrTriggerConnect(ClientConnectionManagerImpl.java:227
 at com.hazelcast.client.connection.nio.ClientConnectionManagerImpl.getOrTriggerConnect(ClientConnectionManagerImpl.java:67
 at com.hazelcast.client.spi.impl.ClientSmartInvocationServiceImpl.getConnection(ClientSmartInvocationServiceImpl.java:78
 at com.hazelcast.client.spi.impl.ClientSmartInvocationServiceImpl.invokeOnPartitionOwner(ClientSmartInvocationServiceImpl.java:49
 at com.hazelcast.client.spi.impl.ClientInvocation.invokeOnSelection(ClientInvocation.java:152
 at com.hazelcast.client.spi.impl.ClientInvocation.invoke(ClientInvocation.java:132
 at com.hazelcast.client.spi.ClientProxy.invokeOnPartition(ClientProxy.java:125
 at com.hazelcast.client.spi.ClientProxy.invoke(ClientProxy.java:120
 at com.hazelcast.client.proxy.ClientMapProxy.tryLock(ClientMapProxy.java:561
 at com.hazelcast.client.proxy.ClientMapProxy.tryLock(ClientMapProxy.java:549
 at com.idea2.distributed.Locks.lock(Locks.java:57
 at com.idea2.distributed.Locks.lock(Locks.java:39
 at com.idea2.distributed.Locks.lock(Locks.java:31
 at com.idea2.shared.logic.integrations.IntegrationsManager.privateDeployEndpoint(IntegrationsManager.java:333
 at com.idea2.shared.logic.integrations.IntegrationsManager.restartEndpoint(IntegrationsManager.java:546
 at com.idea2.shared.logic.integrations.IntegrationsManager.restartEndpoint(IntegrationsManager.java:493
 at com.idea2.apps.logic.refactoring.DataRefactoringService.applyEndpointChange(DataRefactoringService.java:417
 at com.idea2.apps.logic.refactoring.DataRefactoringService.propagateChange(DataRefactoringService.java:406
 at com.idea2.apps.logic.refactoring.DataRefactoringService.applyDataChanges(DataRefactoringService.java:290
 at com.idea2.apps.logic.refactoring.DataRefactoringService.applyChanges(DataRefactoringService.java:261
 at com.idea2.apps.logic.jobs.PushChangesUpdateDataJobExecutor.executeChunk(PushChangesUpdateDataJobExecutor.java:46
 at com.idea2.apps.logic.jobs.framework.BackgroundJobExecutor.wrapExecuteChunk(BackgroundJobExecutor.java:86
 at com.idea2.apps.logic.jobs.AppsJobsExecutor.executeChunk(AppsJobsExecutor.java:78
 at com.idea2.apps.logic.jobs.framework.BackgroundJobChunkExecutionThread.run(BackgroundJobChunkExecutionThread.java:36
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617
 at java.lang.Thread.run(Thread.java:745
 at ------ End remote and begin local stack-trace ------.(Unknown Source
 at com.hazelcast.client.spi.impl.ClientInvocationFuture.resolveResponse(ClientInvocationFuture.java:133
 at com.hazelcast.client.spi.impl.ClientInvocationFuture.get(ClientInvocationFuture.java:100
 at com.hazelcast.client.spi.impl.ClientInvocationFuture.get(ClientInvocationFuture.java:75
 at com.hazelcast.client.spi.impl.ClientInvocationFuture.get(ClientInvocationFuture.java:38
 at com.hazelcast.client.spi.ClientProxy.invokeOnPartition(ClientProxy.java:126
 at com.hazelcast.client.spi.ClientProxy.invoke(ClientProxy.java:120
 at com.hazelcast.client.proxy.ClientMapProxy.tryLock(ClientMapProxy.java:561
 at com.hazelcast.client.proxy.ClientMapProxy.tryLock(ClientMapProxy.java:549
 at com.idea2.distributed.Locks.lock(Locks.java:57
 at com.idea2.distributed.Locks.lock(Locks.java:39
 at com.idea2.distributed.Locks.lock(Locks.java:31
 at com.idea2.shared.logic.integrations.IntegrationsManager.privateDeployEndpoint(IntegrationsManager.java:333
 at com.idea2.shared.logic.integrations.IntegrationsManager.restartEndpoint(IntegrationsManager.java:546
 at com.idea2.shared.logic.integrations.IntegrationsManager.restartEndpoint(IntegrationsManager.java:493
 at com.idea2.apps.logic.refactoring.DataRefactoringService.applyEndpointChange(DataRefactoringService.java:417
 at com.idea2.apps.logic.refactoring.DataRefactoringService.propagateChange(DataRefactoringService.java:406
 at com.idea2.apps.logic.refactoring.DataRefactoringService.applyDataChanges(DataRefactoringService.java:290
 at com.idea2.apps.logic.refactoring.DataRefactoringService.applyChanges(DataRefactoringService.java:261
 at com.idea2.apps.logic.jobs.PushChangesUpdateDataJobExecutor.executeChunk(PushChangesUpdateDataJobExecutor.java:46
 at com.idea2.apps.logic.jobs.framework.BackgroundJobExecutor.wrapExecuteChunk(BackgroundJobExecutor.java:86
 at com.idea2.apps.logic.jobs.AppsJobsExecutor.executeChunk(AppsJobsExecutor.java:78
 at com.idea2.apps.logic.jobs.framework.BackgroundJobChunkExecutionThread.run(BackgroundJobChunkExecutionThread.java:36
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617
 at java.lang.Thread.run(Thread.java:745) Caused by: java.net.ConnectException: Failed to connect to kubernetes.default.svc/10.239.240.1:443 at com.squareup.okhttp.internal.io.RealConnection.connectSocket(RealConnection.java:139
 at com.squareup.okhttp.internal.io.RealConnection.connect(RealConnection.java:108
 at com.squareup.okhttp.internal.http.StreamAllocation.findConnection(StreamAllocation.java:174
 at com.squareup.okhttp.internal.http.StreamAllocation.findHealthyConnection(StreamAllocation.java:126
 at com.squareup.okhttp.internal.http.StreamAllocation.newStream(StreamAllocation.java:95
 at com.squareup.okhttp.internal.http.HttpEngine.connect(HttpEngine.java:283
 at com.squareup.okhttp.internal.http.HttpEngine.sendRequest(HttpEngine.java:224
 at com.squareup.okhttp.Call.getResponse(Call.java:286
 at com.squareup.okhttp.Call$ApplicationInterceptorChain.proceed(Call.java:243
 at io.fabric8.kubernetes.client.utils.HttpClientUtils$3.intercept(HttpClientUtils.java:110
 at com.squareup.okhttp.Call$ApplicationInterceptorChain.proceed(Call.java:232
 at com.squareup.okhttp.Call.getResponseWithInterceptorChain(Call.java:205
 at com.squareup.okhttp.Call.execute(Call.java:80
 at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:203
 at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:196
 at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleGet(BaseOperation.java:483
 at io.fabric8.kubernetes.client.dsl.base.BaseOperation.get(BaseOperation.java Context

Connection failed to Kubernetes when using the service strategy

When Kubernetes runs without the DNS support (as it's a plugin), the resolution fails.

The host is unknown. In this case the URL can be computed as follows: KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT}

See fabric8io/kubernetes-client#35

How to connect a client to a cluster

I have setup my cluster using the service-name option which works fine and multiple hazelcast nodes discover each other. However I can't figure out how to get my clients (spring boot java apps) to connect to all members in the cluster other than by specifying ip address which obviously does not fit with the K8 model. Can somebody advice/add to the docs how to connect a client that is running in the same K8 cluster?

Unable to connect client using kubernetes discovery strategy from xml

I have a hazelcast 3.9.2 running on kubernetes which is using the HazelcastKubernetesDiscoveryStrategy, no issues.

When I try to connect a client using the following xml config.

<hazelcast-client>
  <properties>
    <property name="hazelcast.discovery.enabled">true</property>
  </properties>
  <network>
    <discovery-strategies>
      <discovery-strategy enabled="true"
          class="com.hazelcast.kubernetes.HazelcastKubernetesDiscoveryStrategy">
          <properties>
            <property name="service-name">something</property>
            <property name="namespace">something</property>
          </properties>
      </discovery-strategy>
    </discovery-strategies>
  </network>
</hazelcast-client>

I get the following error.

com.hazelcast.config.InvalidConfigurationException: Invalid configuration
at com.hazelcast.spi.discovery.impl.DefaultDiscoveryService.loadDiscoveryStrategies(DefaultDiscoveryService.java:151)
at com.hazelcast.spi.discovery.impl.DefaultDiscoveryService.<init>(DefaultDiscoveryService.java:60)
at com.hazelcast.spi.discovery.impl.DefaultDiscoveryServiceProvider.newDiscoveryService(DefaultDiscoveryServiceProvider.java:29)
at com.hazelcast.client.impl.HazelcastClientInstanceImpl.initDiscoveryService(HazelcastClientInstanceImpl.java:330)
at com.hazelcast.client.impl.HazelcastClientInstanceImpl.<init>(HazelcastClientInstanceImpl.java:239)
at com.hazelcast.client.HazelcastClient$1.createHazelcastInstanceClient(HazelcastClient.java:55)
at com.hazelcast.client.HazelcastClientManager.newHazelcastClient(HazelcastClientManager.java:74)
at com.hazelcast.client.HazelcastClient.newHazelcastClient(HazelcastClient.java:72)
at CacheConsumer.runApp(CacheConsumer.java:49)
at CacheConsumer.main(CacheConsumer.java:26)
Caused by: com.hazelcast.config.properties.ValidationException: There is no discovery strategy factory to create 'DiscoveryStrategyConfig{properties={namespace=aws-hazelcast, service-name=hazelcast}, className='com.hazelcast.kubernetes.HazelcastKubernetesDiscoveryStrategy', discoveryStrategyFactory=null}' Is it a typo in a strategy classname? Perhaps you forgot to include implementation on a classpath?
at com.hazelcast.spi.discovery.impl.DefaultDiscoveryService.buildDiscoveryStrategy(DefaultDiscoveryService.java:188)
at com.hazelcast.spi.discovery.impl.DefaultDiscoveryService.loadDiscoveryStrategies(DefaultDiscoveryService.java:145)

But when I try the following code it works fine.

DiscoveryConfig dc = config.getNetworkConfig().getDiscoveryConfig();
HazelcastKubernetesDiscoveryStrategyFactory factory = new HazelcastKubernetesDiscoveryStrategyFactory();
DiscoveryStrategyConfig strategyConfig = new DiscoveryStrategyConfig(factory);
strategyConfig.addProperty("service-name","something");
strategyConfig.addProperty("namespace","something");
dc.addDiscoveryStrategyConfig(strategyConfig);

There was a similar issue opened but closed:
#59

The issue suggested possible class path issue, it doesn't seem like a class path issue.
The error message has:

 discoveryStrategyFactory=null

Revise readme

The main readme of this project could get some love. It talks too much about implementation details and too little about actual usage.

Example - this is close to the top of the readme:

While this is at the bottom:

The SPI is purely an implementation detail, while I reckon majority of users are interested in the Docker images.

In general the readme should be more use-case oriented.

Port in DnsEndpointResolver is 0

I just used this module and deployed two services on google cloud, added a headless service which returns service records for both services but the hazelcast connect fails because the DnsEndpointResolver tries to connect to port 0 - the service itself exposes port 5701.

Any thoughts?

Not possible to only provide a service-dns

According to the README it should be possible to provide only a service-dns property when DNS based lookups are required.

However if only this property is provided then it fails to load properties throwing an exception:

https://github.com/noctarius/hazelcast-kubernetes-discovery/blob/hazelcast-kubernetes-discovery-0.9.1/src/main/java/com/noctarius/hazelcast/kubernetes/HazelcastKubernetesDiscoveryStrategy.java#L51

"For kubernetes discovery either 'service-dns' or " + "'service-name' and 'namespace' must be set");
"

To fix, the line should read:

if (serviceDns == null && (serviceName == null || namespace == null)) {

Update copyright license and code formatting

The license in files is outdates (2017).

Project Dependecies

Hi,

I have been using your plugin with great success. 👍
I had less luck in configuring the plugin to use the kubernetes api due to RBAC user permission on the service. However the DNS lookup worked flawlessly and i don't need to anything.

I'm using this version (gradle)

compile group: 'com.hazelcast', name: 'hazelcast-kubernetes', version: '1.1.0'

When gradle resolved the dependecies it also pulls in this:

//https://mvnrepository.com/artifact/com.hazelcast/hazelcast-kubernetes/1.1.0
io.fabric8:kubernetes-client:3.0.3

Looking at the project, this is not used anymore.

Would it be possible to get a new build with this dependency removed?

Regards

ade

Hazelcast starts in Standalone mode in 3.9.2

In the example XML you have

<!-- only necessary prior Hazelcast 3.8 -->
<property name="hazelcast.discovery.enabled">true</property>

This is misleading, we needed to have the property in 3.9 as well. Without it, we'd get a log message saying "No join methods enabled, starting in standalone mode." and the cluster would fail to initialize.

Established socket connection

Why my cluster nodes cannot connect to each other? Every hazelcast instances have the same group name.

Is there a difference for hazelcast and discovery plugin between which kubernetes network do I use?

My configs:

   @Bean
    public Config hazelcastConfig() {
       Config config = new Config();

        config.getGroupConfig()
                .setName("mobiry-cluster")
                .setPassword("password");

        config.setProperty("hazelcast.discovery.enabled", "true");

        JoinConfig joinConfig = config.getNetworkConfig().getJoin();
        joinConfig.getMulticastConfig().setEnabled(false);
        joinConfig.getTcpIpConfig().setEnabled(false);

        DiscoveryConfig discoveryConfig = joinConfig.getDiscoveryConfig();
        HazelcastKubernetesDiscoveryStrategyFactory factory = new HazelcastKubernetesDiscoveryStrategyFactory();

        DiscoveryStrategyConfig strategyConfig = new DiscoveryStrategyConfig(factory);
        strategyConfig.addProperty("service-name", "mobiry");
        strategyConfig.addProperty("service-label-name", "run");
        strategyConfig.addProperty("service-label-value", "mobiry");
        strategyConfig.addProperty("namespace", "default");

        discoveryConfig.addDiscoveryStrategyConfig(strategyConfig);
        return config;
   }

   @Bean
    public WebFilter webFilter(HazelcastInstance hazelcastInstance) {

        Properties properties = new Properties();
        properties.put("instance-name", hazelcastInstance.getName());
        properties.put("sticky-session", "false");

        return new WebFilter(properties);
    }

Logs:

Members [1] {
Member [100.97.75.211]:5701 - 6d545348-b240-40a8-92ed-9214ae00287e this
}
16-Aug-2018 12:04:41.200 INFO [main] com.hazelcast.core.LifecycleService.null [100.97.75.211]:5701 [dev] [3.8.7] [100.97.75.211]:5701 is STARTED
16-Aug-2018 12:04:41.200 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service [Catalina]
16-Aug-2018 12:04:41.200 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet Engine: Apache Tomcat/8.5.32
16-Aug-2018 12:04:41.239 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application archive [/usr/local/tomcat/webapps/ROOT.war]
16-Aug-2018 12:04:42.990 INFO [hz.SESSION-REPLICATION-INSTANCE.IO.thread-Acceptor] com.hazelcast.nio.tcp.SocketAcceptorThread.null [100.97.75.211]:5701 [dev] [3.8.7] Accepting socket connection from /100.112.0.83:38101
16-Aug-2018 12:04:42.999 INFO [hz.SESSION-REPLICATION-INSTANCE.cached.thread-3] com.hazelcast.nio.tcp.TcpIpConnectionManager.null [100.97.75.211]:5701 [dev] [3.8.7] Established socket connection between /100.97.75.211:5701 and /100.112.0.83:38101
16-Aug-2018 12:04:44.515 INFO [localhost-startStop-1] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
16-Aug-2018 12:04:44.524 INFO [localhost-startStop-1] org.apache.catalina.core.ApplicationContext.log No Spring WebApplicationInitializer types detected on classpath
16-Aug-2018 12:04:44.541 INFO [localhost-startStop-1] com.hazelcast.session.HazelcastSessionManager.startInternal contextPath:
16-Aug-2018 12:04:44.557 INFO [localhost-startStop-1] com.hazelcast.session.HazelcastSessionManager.startInternal HazelcastSessionManager started...
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
16-Aug-2018 12:04:44.597 INFO [localhost-startStop-1] org.apache.catalina.core.ApplicationContext.log Initializing Spring FrameworkServlet 'spring-web'
16-Aug-2018 12:04:45.953 INFO [localhost-startStop-1] com.hazelcast.instance.AddressPicker.null [LOCAL] [mobiry-cluster] [3.8.7] Prefer IPv4 stack is true.
16-Aug-2018 12:04:45.957 INFO [localhost-startStop-1] com.hazelcast.instance.AddressPicker.null [LOCAL] [mobiry-cluster] [3.8.7] Picked [100.97.75.211]:5702, using socket ServerSocket[addr=/0:0:0:0:0:0:0:0,localport=5702], bind any local is true
16-Aug-2018 12:04:45.985 INFO [localhost-startStop-1] com.hazelcast.system.null [100.97.75.211]:5702 [mobiry-cluster] [3.8.7] Hazelcast 3.8.7 (20171109 - 0fe6297) starting at [100.97.75.211]:5702
16-Aug-2018 12:04:45.985 INFO [localhost-startStop-1] com.hazelcast.system.null [100.97.75.211]:5702 [mobiry-cluster] [3.8.7] Copyright (c) 2008-2016, Hazelcast, Inc. All Rights Reserved.
16-Aug-2018 12:04:45.985 INFO [localhost-startStop-1] com.hazelcast.system.null [100.97.75.211]:5702 [mobiry-cluster] [3.8.7] Configured Hazelcast Serialization version : 1
16-Aug-2018 12:04:46.202 INFO [localhost-startStop-1] com.hazelcast.spi.impl.operationservice.impl.BackpressureRegulator.null [100.97.75.211]:5702 [mobiry-cluster] [3.8.7] Backpressure is disabled
16-Aug-2018 12:04:46.658 INFO [localhost-startStop-1] com.hazelcast.spi.discovery.integration.DiscoveryService.null [100.97.75.211]:5702 [mobiry-cluster] [3.8.7] Kubernetes Discovery properties: { service-dns: null, service-dns-timeout: 5, service-name: mobiry, service-label: run, service-label-value: mobiry, namespace: default, resolve-not-ready-addresses: false, kubernetes-master: https://kubernetes.default.svc}
16-Aug-2018 12:04:46.659 INFO [localhost-startStop-1] com.hazelcast.spi.discovery.integration.DiscoveryService.null [100.97.75.211]:5702 [mobiry-cluster] [3.8.7] Kubernetes Discovery: Bearer Token { eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImRlZmF1bHQtdG9rZW4teng0aGQiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGVmYXVsdCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6ImUxNzE2ZWM5LWEwOTItMTFlOC05ZGFmLTEyOGZjY2FkYWRiOCIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0OmRlZmF1bHQifQ.2OcyUxuBYHozVSOMrq-ghquLD42nR651Hic_Yqi_UlcpzifryN0O1i3Iz3JBU8MdEanI0lP2olF1HR5k0_DBbqFcX-k1BaiK3AfJtArBd7yn0fDHvJ763MQ14TPeYFNQIk0Eihghh_EhJPHYLhaVCwGz5HSbahvPpf_JvV6uPV6fFRS9mZCUxJCyNj4DV09Xg9Y8i2ZNDncDnhQrvqTKjM0pz3Hwas3ywAs924jHNAn6HXJNxwwD0JjTr_UUFsoYslIW6MtMzOzUdrdtRGx-y0vq7Eu9ACywRgSsxzeNwNt0XqxP6KrbzQQ3-EMQdDIJzBbld_d0pNiG1tLXD7usBQ }
16-Aug-2018 12:04:46.957 INFO [localhost-startStop-1] com.hazelcast.spi.discovery.integration.DiscoveryService.null [100.97.75.211]:5702 [mobiry-cluster] [3.8.7] Kubernetes Discovery activated resolver: ServiceEndpointResolver
16-Aug-2018 12:04:46.958 INFO [localhost-startStop-1] com.hazelcast.spi.discovery.integration.DiscoveryService.null [100.97.75.211]:5702 [mobiry-cluster] [3.8.7] Kubernetes Discovery properties: { service-dns: null, service-dns-timeout: 5, service-name: mobiry, service-label: run, service-label-value: mobiry, namespace: default, resolve-not-ready-addresses: false, kubernetes-master: https://kubernetes.default.svc}
16-Aug-2018 12:04:46.958 INFO [localhost-startStop-1] com.hazelcast.spi.discovery.integration.DiscoveryService.null [100.97.75.211]:5702 [mobiry-cluster] [3.8.7] Kubernetes Discovery: Bearer Token { eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImRlZmF1bHQtdG9rZW4teng0aGQiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGVmYXVsdCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6ImUxNzE2ZWM5LWEwOTItMTFlOC05ZGFmLTEyOGZjY2FkYWRiOCIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0OmRlZmF1bHQifQ.2OcyUxuBYHozVSOMrq-ghquLD42nR651Hic_Yqi_UlcpzifryN0O1i3Iz3JBU8MdEanI0lP2olF1HR5k0_DBbqFcX-k1BaiK3AfJtArBd7yn0fDHvJ763MQ14TPeYFNQIk0Eihghh_EhJPHYLhaVCwGz5HSbahvPpf_JvV6uPV6fFRS9mZCUxJCyNj4DV09Xg9Y8i2ZNDncDnhQrvqTKjM0pz3Hwas3ywAs924jHNAn6HXJNxwwD0JjTr_UUFsoYslIW6MtMzOzUdrdtRGx-y0vq7Eu9ACywRgSsxzeNwNt0XqxP6KrbzQQ3-EMQdDIJzBbld_d0pNiG1tLXD7usBQ }
16-Aug-2018 12:04:46.976 INFO [localhost-startStop-1] com.hazelcast.spi.discovery.integration.DiscoveryService.null [100.97.75.211]:5702 [mobiry-cluster] [3.8.7] Kubernetes Discovery activated resolver: ServiceEndpointResolver
16-Aug-2018 12:04:46.976 INFO [localhost-startStop-1] com.hazelcast.instance.Node.null [100.97.75.211]:5702 [mobiry-cluster] [3.8.7] Activating Discovery SPI Joiner
16-Aug-2018 12:04:47.146 INFO [localhost-startStop-1] com.hazelcast.spi.impl.operationexecutor.impl.OperationExecutorImpl.null [100.97.75.211]:5702 [mobiry-cluster] [3.8.7] Starting 2 partition threads
16-Aug-2018 12:04:47.162 INFO [localhost-startStop-1] com.hazelcast.spi.impl.operationexecutor.impl.OperationExecutorImpl.null [100.97.75.211]:5702 [mobiry-cluster] [3.8.7] Starting 3 generic threads (1 dedicated for priority tasks)
16-Aug-2018 12:04:47.176 INFO [localhost-startStop-1] com.hazelcast.core.LifecycleService.null [100.97.75.211]:5702 [mobiry-cluster] [3.8.7] [100.97.75.211]:5702 is STARTING
16-Aug-2018 12:04:47.610 INFO [hz._hzInstance_1_mobiry-cluster.cached.thread-3] com.hazelcast.nio.tcp.InitConnectionTask.null [100.97.75.211]:5702 [mobiry-cluster] [3.8.7] Connecting to /100.97.75.211:5701, timeout: 0, bind-any: true
16-Aug-2018 12:04:47.611 INFO [hz.SESSION-REPLICATION-INSTANCE.IO.thread-Acceptor] com.hazelcast.nio.tcp.SocketAcceptorThread.null [100.97.75.211]:5701 [dev] [3.8.7] Accepting socket connection from /100.97.75.211:58316
16-Aug-2018 12:04:47.612 INFO [hz.SESSION-REPLICATION-INSTANCE.cached.thread-2] com.hazelcast.nio.tcp.TcpIpConnectionManager.null [100.97.75.211]:5701 [dev] [3.8.7] Established socket connection between /100.97.75.211:5701 and /100.97.75.211:58316
16-Aug-2018 12:04:47.622 INFO [hz._hzInstance_1_mobiry-cluster.cached.thread-3] com.hazelcast.nio.tcp.TcpIpConnectionManager.null [100.97.75.211]:5702 [mobiry-cluster] [3.8.7] Established socket connection between /100.97.75.211:58316 and /100.97.75.211:5701
16-Aug-2018 12:04:47.624 INFO [hz._hzInstance_1_mobiry-cluster.cached.thread-2] com.hazelcast.nio.tcp.InitConnectionTask.null [100.97.75.211]:5702 [mobiry-cluster] [3.8.7] Connecting to /100.112.0.83:5701, timeout: 0, bind-any: true
16-Aug-2018 12:04:47.625 INFO [hz._hzInstance_1_mobiry-cluster.cached.thread-2] com.hazelcast.nio.tcp.TcpIpConnectionManager.null [100.97.75.211]:5702 [mobiry-cluster] [3.8.7] Established socket connection between /100.97.75.211:49449 and /100.112.0.83:5701
16-Aug-2018 12:04:48.632 WARNING [hz.SESSION-REPLICATION-INSTANCE.priority-generic-operation.thread-0] com.hazelcast.internal.cluster.impl.ClusterJoinManager.null [100.97.75.211]:5701 [dev] [3.8.7] Received an invalid join request from [100.97.75.211]:5702, cause: clusters part of different cluster-groups
16-Aug-2018 12:04:48.634 WARNING [hz.SESSION-REPLICATION-INSTANCE.priority-generic-operation.thread-0] com.hazelcast.internal.cluster.impl.ClusterJoinManager.null [100.97.75.211]:5701 [dev] [3.8.7] Received an invalid join request from [100.97.75.211]:5702, cause: clusters part of different cluster-groups
16-Aug-2018 12:04:48.636 INFO [hz._hzInstance_1_mobiry-cluster.generic-operation.thread-1] com.hazelcast.nio.tcp.TcpIpConnection.null [100.97.75.211]:5702 [mobiry-cluster] [3.8.7] Connection[id=1, /100.97.75.211:58316->/100.97.75.211:5701, endpoint=[100.97.75.211]:5701, alive=false, type=MEMBER] closed. Reason: Node could not join cluster at node: [100.97.75.211]:5701 Cause: the target cluster has a different group-name
16-Aug-2018 12:04:48.639 INFO [hz._hzInstance_1_mobiry-cluster.generic-operation.thread-0] com.hazelcast.nio.tcp.TcpIpConnection.null [100.97.75.211]:5702 [mobiry-cluster] [3.8.7] Connection[id=2, /100.97.75.211:49449->/100.112.0.83:5701, endpoint=[100.112.0.83]:5701, alive=false, type=MEMBER] closed. Reason: Node could not join cluster at node: [100.112.0.83]:5701 Cause: the target cluster has a different group-name
16-Aug-2018 12:04:48.639 WARNING [hz._hzInstance_1_mobiry-cluster.priority-generic-operation.thread-0] com.hazelcast.cluster.null [100.97.75.211]:5702 [mobiry-cluster] [3.8.7] Node could not join cluster at node: [100.97.75.211]:5701 Cause: the target cluster has a different group-name
16-Aug-2018 12:04:48.640 INFO [hz._hzInstance_1_mobiry-cluster.priority-generic-operation.thread-0] com.hazelcast.internal.cluster.impl.DiscoveryJoiner.null [100.97.75.211]:5702 [mobiry-cluster] [3.8.7] [100.97.75.211]:5701 is added to the blacklist.
16-Aug-2018 12:04:48.640 WARNING [hz._hzInstance_1_mobiry-cluster.priority-generic-operation.thread-0] com.hazelcast.cluster.null [100.97.75.211]:5702 [mobiry-cluster] [3.8.7] Node could not join cluster at node: [100.112.0.83]:5701 Cause: the target cluster has a different group-name
16-Aug-2018 12:04:48.640 INFO [hz._hzInstance_1_mobiry-cluster.priority-generic-operation.thread-0] com.hazelcast.internal.cluster.impl.DiscoveryJoiner.null [100.97.75.211]:5702 [mobiry-cluster] [3.8.7] [100.112.0.83]:5701 is added to the blacklist.
16-Aug-2018 12:04:48.640 INFO [localhost-startStop-1] com.hazelcast.system.null [100.97.75.211]:5702 [mobiry-cluster] [3.8.7] Cluster version set to 3.8
16-Aug-2018 12:04:48.641 INFO [hz.SESSION-REPLICATION-INSTANCE.IO.thread-in-1] com.hazelcast.nio.tcp.TcpIpConnection.null [100.97.75.211]:5701 [dev] [3.8.7] Connection[id=2, /100.97.75.211:5701->/100.97.75.211:58316, endpoint=[100.97.75.211]:5702, alive=false, type=MEMBER] closed. Reason: Connection closed by the other side
16-Aug-2018 12:04:48.642 INFO [localhost-startStop-1] com.hazelcast.internal.cluster.impl.DiscoveryJoiner.null [100.97.75.211]:5702 [mobiry-cluster] [3.8.7]
Members [1] {
Member [100.97.75.211]:5702 - 4c9e1694-6ba5-4079-a9c0-741a08040a0f this
}
16-Aug-2018 12:04:48.644 WARNING [hz._hzInstance_1_mobiry-cluster.generic-operation.thread-1] com.hazelcast.cluster.null [100.97.75.211]:5702 [mobiry-cluster] [3.8.7] Node could not join cluster at node: [100.97.75.211]:5701 Cause: the target cluster has a different group-name
16-Aug-2018 12:04:48.644 INFO [hz._hzInstance_1_mobiry-cluster.generic-operation.thread-1] com.hazelcast.internal.cluster.impl.DiscoveryJoiner.null [100.97.75.211]:5702 [mobiry-cluster] [3.8.7] [100.97.75.211]:5701 is added to the blacklist.
16-Aug-2018 12:04:48.643 WARNING [hz._hzInstance_1_mobiry-cluster.generic-operation.thread-0] com.hazelcast.cluster.null [100.97.75.211]:5702 [mobiry-cluster] [3.8.7] Node could not join cluster at node: [100.112.0.83]:5701 Cause: the target cluster has a different group-name
16-Aug-2018 12:04:48.644 INFO [hz._hzInstance_1_mobiry-cluster.generic-operation.thread-0] com.hazelcast.internal.cluster.impl.DiscoveryJoiner.null [100.97.75.211]:5702 [mobiry-cluster] [3.8.7] [100.112.0.83]:5701 is added to the blacklist.
16-Aug-2018 12:04:48.644 WARNING [localhost-startStop-1] com.hazelcast.instance.Node.null [100.97.75.211]:5702 [mobiry-cluster] [3.8.7] Config seed port is 5701 and cluster size is 1. Some of the ports seem occupied!
16-Aug-2018 12:04:48.680 INFO [localhost-startStop-1] com.hazelcast.core.LifecycleService.null [100.97.75.211]:5702 [mobiry-cluster] [3.8.7] [100.97.75.211]:5702 is STARTED
16-Aug-2018 12:04:48.779 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deployment of web application archive [/usr/local/tomcat/webapps/ROOT.war] has finished in [7,539] ms
16-Aug-2018 12:04:48.782 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-80"]
16-Aug-2018 12:04:48.789 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["ajp-nio-8009"]
16-Aug-2018 12:04:48.791 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in 11191 ms
16-Aug-2018 12:09:47.142 INFO [hz._hzInstance_1_mobiry-cluster.cached.thread-1] com.hazelcast.nio.tcp.TcpIpConnectionManager.null [100.97.75.211]:5702 [mobiry-cluster] [3.8.7] Established socket connection between /100.97.75.211:56246 and /100.97.75.211:5701
16-Aug-2018 12:09:47.142 INFO [hz.SESSION-REPLICATION-INSTANCE.IO.thread-Acceptor] com.hazelcast.nio.tcp.SocketAcceptorThread.null [100.97.75.211]:5701 [dev] [3.8.7] Accepting socket connection from /100.97.75.211:56246
16-Aug-2018 12:09:47.142 INFO [hz.SESSION-REPLICATION-INSTANCE.cached.thread-3] com.hazelcast.nio.tcp.TcpIpConnectionManager.null [100.97.75.211]:5701 [dev] [3.8.7] Established socket connection between /100.97.75.211:5701 and /100.97.75.211:56246
16-Aug-2018 12:09:47.164 INFO [hz._hzInstance_1_mobiry-cluster.cached.thread-1] com.hazelcast.nio.tcp.TcpIpConnectionManager.null [100.97.75.211]:5702 [mobiry-cluster] [3.8.7] Established socket connection between /100.97.75.211:43313 and /100.112.0.83:5701

Is it working properly?

I think there should be more information about how this discovery plugin suppose to work inside Kubernetes. At the moment I can see the following logic:

Plugin queries either DNS or REST for Service endpoint (I have used DNS)
It gets Service IP + port
Tries to establish connection with kubernetes service that randomly forwards query to backend hazelcast replicas in order to form the cluster
??? fails actually, because service itself is ephemeral and is not a member of the cluster

Here are logs from one of the running pods inside minikube where I have tried to combine it with management server:

$ kubectl logs hazelcast-server-d978ff479-vx4kk
Kubernetes Namespace: default
Kubernetes Service DNS:
'########################################
'# RUN_JAVA=
'# JAVA_OPTS=-Xms512M -Xmx2056M -Dcom.sun.management.jmxremote -'Dcom.sun.management.jmxremote.authenticate=false -Dhazelcast.diagnostics.enabled=true -'Dhazelcast.rest.enabled=true
'# CLASSPATH=/:/opt/hazelcast/:/opt/hazelcast/external/*:
'########################################
Checking custom configuration
no custom configuration found
Mar 06, 2018 6:47:52 AM com.hazelcast.config.XmlConfigLocator
INFO: Loading 'hazelcast.xml' from working directory.
Mar 06, 2018 6:47:52 AM com.hazelcast.instance.AddressPicker
INFO: [LOCAL] [dev] [3.9.3] Prefer IPv4 stack is true.
Mar 06, 2018 6:47:52 AM com.hazelcast.instance.AddressPicker
INFO: [LOCAL] [dev] [3.9.3] Picked [172.17.0.7]:5701, using socket ServerSocket[addr=/0.0.0.0,localport=5701], bind any local is true
Mar 06, 2018 6:47:53 AM com.hazelcast.system
INFO: [172.17.0.7]:5701 [dev] [3.9.3] Hazelcast 3.9.3 (20180216 - 539b124) starting at [172.17.0.7]:5701
Mar 06, 2018 6:47:53 AM com.hazelcast.system
INFO: [172.17.0.7]:5701 [dev] [3.9.3] Copyright (c) 2008-2018, Hazelcast, Inc. All Rights Reserved.
Mar 06, 2018 6:47:53 AM com.hazelcast.system
INFO: [172.17.0.7]:5701 [dev] [3.9.3] Configured Hazelcast Serialization version: 1
Mar 06, 2018 6:47:53 AM com.hazelcast.spi.impl.operationservice.impl.BackpressureRegulator
INFO: [172.17.0.7]:5701 [dev] [3.9.3] Backpressure is disabled
Mar 06, 2018 6:47:53 AM com.hazelcast.spi.discovery.integration.DiscoveryService
INFO: [172.17.0.7]:5701 [dev] [3.9.3] Kubernetes Discovery properties: { service-dns: hazelcast-service.default.svc.cluster.local, service-dns-timeout: 5, service-name: null, service-label: null, service-label-value: true, namespace: default, resolve-not-ready-addresses: false, kubernetes-master: https://kubernetes.default.svc}
Mar 06, 2018 6:47:53 AM com.hazelcast.spi.discovery.integration.DiscoveryService
INFO: [172.17.0.7]:5701 [dev] [3.9.3] Kubernetes Discovery activated resolver: DnsEndpointResolver
Mar 06, 2018 6:47:53 AM com.hazelcast.instance.Node
INFO: [172.17.0.7]:5701 [dev] [3.9.3] Activating Discovery SPI Joiner
Mar 06, 2018 6:47:54 AM com.hazelcast.spi.impl.operationexecutor.impl.OperationExecutorImpl
INFO: [172.17.0.7]:5701 [dev] [3.9.3] Starting 2 partition threads and 3 generic threads (1 dedicated for priority tasks)
Mar 06, 2018 6:47:54 AM com.hazelcast.internal.diagnostics.Diagnostics
INFO: [172.17.0.7]:5701 [dev] [3.9.3] Diagnostics started
Mar 06, 2018 6:47:54 AM com.hazelcast.internal.diagnostics.BuildInfoPlugin
INFO: [172.17.0.7]:5701 [dev] [3.9.3] Plugin:active
Mar 06, 2018 6:47:54 AM com.hazelcast.internal.diagnostics.SystemPropertiesPlugin
INFO: [172.17.0.7]:5701 [dev] [3.9.3] Plugin:active
Mar 06, 2018 6:47:54 AM com.hazelcast.internal.diagnostics.ConfigPropertiesPlugin
INFO: [172.17.0.7]:5701 [dev] [3.9.3] Plugin:active
Mar 06, 2018 6:47:54 AM com.hazelcast.internal.diagnostics.MetricsPlugin
INFO: [172.17.0.7]:5701 [dev] [3.9.3] Plugin:active, period-millis:60000
Mar 06, 2018 6:47:54 AM com.hazelcast.internal.diagnostics.SlowOperationPlugin
INFO: [172.17.0.7]:5701 [dev] [3.9.3] Plugin:active, period-millis:60000
Mar 06, 2018 6:47:54 AM com.hazelcast.internal.diagnostics.MemberHazelcastInstanceInfoPlugin
INFO: [172.17.0.7]:5701 [dev] [3.9.3] Plugin:active, period-millis:60000
Mar 06, 2018 6:47:54 AM com.hazelcast.internal.diagnostics.SystemLogPlugin
INFO: [172.17.0.7]:5701 [dev] [3.9.3] Plugin:active: logPartitions:false
Mar 06, 2018 6:47:54 AM com.hazelcast.internal.diagnostics.MemberHazelcastInstanceInfoPlugin
INFO: [172.17.0.7]:5701 [dev] [3.9.3] Plugin:active, period-millis:10000
Mar 06, 2018 6:47:54 AM com.hazelcast.internal.diagnostics.OperationHeartbeatPlugin
INFO: [172.17.0.7]:5701 [dev] [3.9.3] Plugin:active: period-millis:10000 max-deviation:33%
Mar 06, 2018 6:47:54 AM com.hazelcast.core.LifecycleService
INFO: [172.17.0.7]:5701 [dev] [3.9.3] [172.17.0.7]:5701 is STARTING
Mar 06, 2018 6:47:54 AM com.hazelcast.nio.tcp.TcpIpConnector
INFO: [172.17.0.7]:5701 [dev] [3.9.3] Connecting to /10.99.49.133:5701, timeout: 0, bind-any: true
Mar 06, 2018 6:47:55 AM com.hazelcast.nio.tcp.TcpIpConnectionManager
INFO: [172.17.0.7]:5701 [dev] [3.9.3] Established socket connection between /172.17.0.7:44347 and /10.99.49.133:5701
Mar 06, 2018 6:47:55 AM com.hazelcast.nio.tcp.TcpIpConnection
INFO: [172.17.0.7]:5701 [dev] [3.9.3] Connection[id=1, /172.17.0.7:44347->/10.99.49.133:5701, endpoint=[10.99.49.133]:5701, alive=false, type=MEMBER] closed. Reason: Connection closed by the other side
Mar 06, 2018 6:47:56 AM com.hazelcast.nio.tcp.TcpIpConnector
INFO: [172.17.0.7]:5701 [dev] [3.9.3] Connecting to /10.99.49.133:5701, timeout: 0, bind-any: true
Mar 06, 2018 6:47:57 AM com.hazelcast.nio.tcp.TcpIpConnectionManager
INFO: [172.17.0.7]:5701 [dev] [3.9.3] Established socket connection between /172.17.0.7:51935 and /10.99.49.133:5701
Mar 06, 2018 6:47:57 AM com.hazelcast.nio.tcp.TcpIpConnection
INFO: [172.17.0.7]:5701 [dev] [3.9.3] Connection[id=2, /172.17.0.7:51935->/10.99.49.133:5701, endpoint=[10.99.49.133]:5701, alive=false, type=MEMBER] closed. Reason: Connection closed by the other side
Mar 06, 2018 6:47:58 AM com.hazelcast.nio.tcp.TcpIpConnector
INFO: [172.17.0.7]:5701 [dev] [3.9.3] Connecting to /10.99.49.133:5701, timeout: 0, bind-any: true
Mar 06, 2018 6:47:59 AM com.hazelcast.system
INFO: [172.17.0.7]:5701 [dev] [3.9.3] Cluster version set to 3.9
Mar 06, 2018 6:47:59 AM com.hazelcast.internal.cluster.ClusterService
INFO: [172.17.0.7]:5701 [dev] [3.9.3]

Members {size:1, ver:1} [
Member [172.17.0.7]:5701 - 1cded4bf-6f7b-4bf8-92cc-c11645db87dc this
]

Mar 06, 2018 6:47:59 AM com.hazelcast.internal.management.ManagementCenterService
INFO: [172.17.0.7]:5701 [dev] [3.9.3] Hazelcast will connect to Hazelcast Management Center on address:
http://localhost:8080/mancenter
Mar 06, 2018 6:47:59 AM com.hazelcast.core.LifecycleService
INFO: [172.17.0.7]:5701 [dev] [3.9.3] [172.17.0.7]:5701 is STARTED
Mar 06, 2018 6:47:59 AM com.hazelcast.internal.management.ManagementCenterService
INFO: [172.17.0.7]:5701 [dev] [3.9.3] Failed to pull tasks from management center
Mar 06, 2018 6:47:59 AM com.hazelcast.internal.management.ManagementCenterService
INFO: [172.17.0.7]:5701 [dev] [3.9.3] Failed to connect to:http://localhost:8080/mancenter/collector.do
Mar 06, 2018 6:48:01 AM com.hazelcast.nio.tcp.TcpIpConnectionManager
INFO: [172.17.0.7]:5701 [dev] [3.9.3] Established socket connection between /172.17.0.7:60675 and /10.99.49.133:5701
Mar 06, 2018 6:48:01 AM com.hazelcast.nio.tcp.TcpIpConnection
INFO: [172.17.0.7]:5701 [dev] [3.9.3] Connection[id=3, /172.17.0.7:60675->/10.99.49.133:5701, endpoint=[10.99.49.133]:5701, alive=false, type=MEMBER] closed. Reason: Connection closed by the other side

Q: Given scenario of changing replicas from 0 to 2+ ... members join/discover each other later than desired

Hi,
Thanks for the plugin, it is working really well.

One scenario I have noticed is that when there are no running pods with Hazelcast and we try to start with replicas of 2 or more (jump from 0 to 2+ replicas). In this scenario the initial discovery does not find any currently running pods to join and so the 2 replicas start without joining each other.

At some point later they join/discover each other. So it seems (and makes sense) that Hazelcast triggers/polls the discovery plugin periodically to find new nodes etc.

Now in our use case we are using a replicated map and we need to be careful to avoid this scenario (going from 0 pods to many pods) as the replicated maps are not in sync just at the point of pod startup and that is a problem for our use case. The replicated maps do come into sync at some point later (I believe Hazelcast polls the discovery plugin later and the nodes join and sync).

I wonder if others have looked at this and got other thoughts.

Cheers, Rob.

Discovery issue when doing blue green rolling updates

Hello team,

I'm not sure if this is the correct place for this issue, but I believe its atleast a starting point, so please do let me know if I was wrong to create it here.

We use Hazelcast as 2nd level cache for Hibernate and I'm facing a weird issue, during a blue-green deployment scenario below with Kubernetes and I doubt if it is due to the discovery strategy, somehow I have never faced a similar issue with Hazelcast in the past with multicast and with Eureka discovery as we use it extensively in JHipster.

The application is clustered and runs in Pod A and Pod B with a load balancer
The application is at V1 for example
We deploy V2 of application, which has new fields added to an entity class, to Pod A
V2 successfully deploys in Pod A and updates the DB with new schema changes
Somehow app in Pod B with V1 replicates its cache to Pod A which is in V2
Now we deploy Pod B with V2 successfully
Now app in Pod A which has V2 of the app but incorrect cache(v1) replicates to Pod B and hence both pod has old cache data
Now when you access the entity Hibernate fails with below stack trace

Can this be solved in the discovery strategy so that newer versions do not try to replicate from an older version of the app? Is my understanding wrong?

The below is the stack trace from Hibernate when it reads from the cache as the cache is outdated now

java.lang.ArrayIndexOutOfBoundsException: 11
	at org.hibernate.type.TypeHelper.assemble(TypeHelper.java:96)
	at org.hibernate.cache.spi.entry.StandardCacheEntryImpl.assemble(StandardCacheEntryImpl.java:141)
	at org.hibernate.event.internal.DefaultLoadEventListener.convertCacheEntryToEntity(DefaultLoadEventListener.java:754)
	at org.hibernate.event.internal.DefaultLoadEventListener.processCachedEntry(DefaultLoadEventListener.java:625)
	at org.hibernate.event.internal.DefaultLoadEventListener.loadFromSecondLevelCache(DefaultLoadEventListener.java:602)
	at org.hibernate.event.internal.DefaultLoadEventListener.doLoad(DefaultLoadEventListener.java:462)
	at org.hibernate.event.internal.DefaultLoadEventListener.load(DefaultLoadEventListener.java:219)
	at org.hibernate.event.internal.DefaultLoadEventListener.proxyOrLoad(DefaultLoadEventListener.java:278)
	at org.hibernate.event.internal.DefaultLoadEventListener.doOnLoad(DefaultLoadEventListener.java:121)
	at org.hibernate.event.internal.DefaultLoadEventListener.onLoad(DefaultLoadEventListener.java:89)
	at org.hibernate.internal.SessionImpl.fireLoad(SessionImpl.java:1237)
	at org.hibernate.internal.SessionImpl.internalLoad(SessionImpl.java:1120)
	at org.hibernate.type.EntityType.resolveIdentifier(EntityType.java:646)
	at org.hibernate.type.EntityType.resolve(EntityType.java:431)
	at org.hibernate.engine.internal.TwoPhaseLoad.doInitializeEntity(TwoPhaseLoad.java:165)
	at org.hibernate.engine.internal.TwoPhaseLoad.initializeEntity(TwoPhaseLoad.java:125)
	at org.hibernate.loader.Loader.initializeEntitiesAndCollections(Loader.java:1146)
	at org.hibernate.loader.Loader.processResultSet(Loader.java:1005)
	at org.hibernate.loader.Loader.doQuery(Loader.java:943)
	at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:349)
	at org.hibernate.loader.Loader.doList(Loader.java:2615)
	at org.hibernate.loader.Loader.doList(Loader.java:2598)
	at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2430)
	at org.hibernate.loader.Loader.list(Loader.java:2425)
	at org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:502)
	at org.hibernate.hql.internal.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:379)
	at org.hibernate.engine.query.spi.HQLQueryPlan.performList(HQLQueryPlan.java:216)
	at org.hibernate.internal.SessionImpl.list(SessionImpl.java:1488)
	at org.hibernate.query.internal.AbstractProducedQuery.doList(AbstractProducedQuery.java:1445)
	at org.hibernate.query.internal.AbstractProducedQuery.list(AbstractProducedQuery.java:1414)
	at org.hibernate.query.Query.getResultList(Query.java:146)
	at org.hibernate.query.criteria.internal.compile.CriteriaQueryTypeQueryAdapter.getResultList(CriteriaQueryTypeQueryAdapter.java:72)
	at org.springframework.data.jpa.repository.support.SimpleJpaRepository.findAll(SimpleJpaRepository.java:329)
	at org.springframework.data.jpa.repository.support.SimpleJpaRepository.findAll(SimpleJpaRepository.java:74)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.data.repository.core.support.RepositoryFactorySupport$QueryExecutorMethodInterceptor.executeMethodOn(RepositoryFactorySupport.java:504)
	at org.springframework.data.repository.core.support.RepositoryFactorySupport$QueryExecutorMethodInterceptor.doInvoke(RepositoryFactorySupport.java:489)
	at org.springframework.data.repository.core.support.RepositoryFactorySupport$QueryExecutorMethodInterceptor.invoke(RepositoryFactorySupport.java:461)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.data.projection.DefaultMethodInvokingMethodInterceptor.invoke(DefaultMethodInvokingMethodInterceptor.java:56)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99)
	at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:282)
	at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:136)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.data.jpa.repository.support.CrudMethodMetadataPostProcessor$CrudMethodMetadataPopulatingMethodInterceptor.invoke(CrudMethodMetadataPostProcessor.java:133)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.data.repository.core.support.SurroundingTransactionDetectorMethodInterceptor.invoke(SurroundingTransactionDetectorMethodInterceptor.java:57)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
	at com.sun.proxy.$Proxy183.findAll(Unknown Source)
	at com.xxx.xxx.service.DashboardService.findAll(DashboardService.java:89)
	at com.xxx.xxx.service.DashboardService$$FastClassBySpringCGLIB$$4c9a082d.invoke(<generated>)
	at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
	at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99)
	at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:282)
	at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673)
	at com.xxx.xxx.service.DashboardService$$EnhancerBySpringCGLIB$$6bdfcfa9.findAll(<generated>)
	at com.xebialabs.agatha.web.rest.ProjectDashboardResource.getAllProjectDashboards(ProjectDashboardResource.java:95)
	at com.xebialabs.agatha.web.rest.ProjectDashboardResource$$FastClassBySpringCGLIB$$180b3b55.invoke(<generated>)
	at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
	at com.ryantenney.metrics.spring.TimedMethodInterceptor.invoke(TimedMethodInterceptor.java:48)
	at com.ryantenney.metrics.spring.TimedMethodInterceptor.invoke(TimedMethodInterceptor.java:34)
	at com.ryantenney.metrics.spring.AbstractMetricMethodInterceptor.invoke(AbstractMetricMethodInterceptor.java:59)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673)
	at com.xebialabs.agatha.web.rest.ProjectDashboardResource$$EnhancerBySpringCGLIB$$bb71fab6.getAllProjectDashboards(<generated>)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
	at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
	at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
	at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
	at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
	at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
	at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85)
	at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129)
	at com.codahale.metrics.servlet.AbstractInstrumentedFilter.doFilter(AbstractInstrumentedFilter.java:111)
	at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
	at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
	at org.springframework.boot.web.filter.ApplicationContextHeaderFilter.doFilterInternal(ApplicationContextHeaderFilter.java:55)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
	at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:101)
	at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
	at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
	at org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:110)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
	at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)
	at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127)
	at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:114)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:170)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	at com.xebialabs.agatha.security.jwt.JWTFilter.doFilter(JWTFilter.java:43)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	at org.springframework.web.filter.CorsFilter.doFilterInternal(CorsFilter.java:96)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:116)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:64)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	at org.springframework.security.web.access.channel.ChannelProcessingFilter.doFilter(ChannelProcessingFilter.java:157)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)
	at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177)
	at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346)
	at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:262)
	at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
	at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
	at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
	at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
	at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:108)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
	at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
	at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:81)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
	at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
	at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
	at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
	at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)
	at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
	at io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:64)
	at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
	at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131)
	at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
	at io.undertow.security.handlers.AuthenticationConstraintHandler.handleRequest(AuthenticationConstraintHandler.java:53)
	at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
	at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
	at io.undertow.servlet.handlers.security.ServletSecurityConstraintHandler.handleRequest(ServletSecurityConstraintHandler.java:59)
	at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
	at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
	at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
	at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292)
	at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
	at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
	at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
	at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
	at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
	at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
	at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
	at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
	at io.undertow.server.Connectors.executeRootHandler(Connectors.java:332)
	at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:812)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

Include Hazelcast Kubernetes in hazelcast-all

ClassNotFoundException: io.fabric8.kubernetes.client.EditableConfig

Hi trying to deploy a cluster of Hazelcast in k8s using this library.

My understanding is that I need to build a new Docker image for Hazelcast that contains this jar file and a new config and deploy it in k8s.

So I did the following:

change the Dockerfile

FROM openjdk:8-jre
ENV HZ_VERSION 3.9
...
ADD http://central.maven.org/maven2/com/hazelcast/hazelcast-kubernetes/1.0.0/hazelcast-kubernetes-1.0.0.jar $HZ_HOME
ADD http://central.maven.org/maven2/io/fabric8/kubernetes-client/3.1.0/kubernetes-client-3.1.0.jar $HZ_HOME
ADD hazelcast.xml $HZ_HOME

add the jar to the classpath in server.sh

export CLASSPATH=$HAZELCAST_HOME/hazelcast-kubernetes-1.0.0.jar:$CLASSPATH
export CLASSPATH=$HAZELCAST_HOME/kubernetes-client-3.1.0.jar:$CLASSPATH

and use the config hazelcast.xml

<hazelcast xsi:schemaLocation="http://www.hazelcast.com/schema/config hazelcast-config-2.0.xsd"
    xmlns="http://www.hazelcast.com/schema/config"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
    <properties>
    </properties>

    <network>
        <join>
            <!-- deactivate normal discovery -->
            <multicast enabled="false"/>
            <tcp-ip enabled="false"/>

            <!-- activate the Kubernetes plugin -->
            <discovery-strategies>
                <discovery-strategy enabled="true"
                                    class="com.hazelcast.kubernetes.HazelcastKubernetesDiscoveryStrategy">
                    <properties>
                        <!-- configure discovery service API lookup -->
                        <property name="service-name">${hazelcast.k8s.service}</property>
                        <property name="namespace">${hazelcast.k8s.namespace}</property>
                    </properties>
                </discovery-strategy>
            </discovery-strategies>
        </join>
    </network>
</hazelcast>

When the pod start I get the logging message:

Kubernetes Discovery properties: { service-dns: null, service-dns-timeout: 5, service-name: hazelcast, service-label: null, service-label-value: true, namespace: test, kubernetes-master: https://kubernetes.default.svc}

but then just after that I get:

Exception in thread "main" java.lang.NoClassDefFoundError: io/fabric8/kubernetes/client/EditableConfig
	at com.hazelcast.kubernetes.HazelcastKubernetesDiscoveryStrategy.&lt;init&gt;(HazelcastKubernetesDiscoveryStrategy.java:75)
	at com.hazelcast.kubernetes.HazelcastKubernetesDiscoveryStrategyFactory.newDiscoveryStrategy(HazelcastKubernetesDiscoveryStrategyFactory.java:56)

What I cannot understand is that EditableConfig is an old deprecated class from kubernetes-client. I'm using the latest version of the client so I don't understand how the old client would get into the classpath

Am I doing the correct thing here? If so any idea how this exception could happen?

NPE received during Hazelcast start up

During initialization of Hazelcast server we got the following exception:

java.lang.NullPointerException at com.noctarius.hazelcast.kubernetes.ServiceEndpointResolver.getDiscoveryNodes(ServiceEndpointResolver.java:95 at com.noctarius.hazelcast.kubernetes.ServiceEndpointResolver.resolve(ServiceEndpointResolver.java:75 at com.noctarius.hazelcast.kubernetes.HazelcastKubernetesDiscoveryStrategy.discoverNodes(HazelcastKubernetesDiscoveryStrategy.java:88 at com.hazelcast.spi.discovery.impl.DefaultDiscoveryService.discoverNodes(DefaultDiscoveryService.java:74 at com.hazelcast.client.spi.impl.discovery.DiscoveryAddressProvider.loadAddresses(DiscoveryAddressProvider.java:44 at com.hazelcast.client.spi.impl.ClusterListenerSupport.getSocketAddresses(ClusterListenerSupport.java:174 at com.hazelcast.client.spi.impl.ClusterListenerSupport.connect(ClusterListenerSupport.java:233 at com.hazelcast.client.spi.impl.ClusterListenerSupport.connectToOne(ClusterListenerSupport.java:209 at com.hazelcast.client.spi.impl.ClusterListenerSupport.connectToCluster(ClusterListenerSupport.java:161 at com.hazelcast.client.spi.impl.ClientClusterServiceImpl.start(ClientClusterServiceImpl.java:203 at com.hazelcast.client.impl.HazelcastClientInstanceImpl.start(HazelcastClientInstanceImpl.java:303 at com.idea2.distributed.DistributedServices.restart(DistributedServices.java:311 at com.idea2.distributed.exceptions.ExceptionsUtils.rethrow(ExceptionsUtils.java:38 at com.idea2.distributed.exceptions.ExceptionsUtils.manageError(ExceptionsUtils.java:26 at com.idea2.distributed.Queues.getQueue(Queues.java:24 at com.idea2.platformmanager.logic.tasks.framework.TasksListener.lambda$0(TasksListener.java:45 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617 at java.lang.Thread.run(Thread.java:745) Context

The service config that it is used for discover is:

Name: hazelcast Namespace: default Labels: <none> Selector: type=hazelcast Type: ClusterIP IP: 10.239.255.186 Port: hazel 5701/TCP Endpoints: <none> Session Affinity: None No events.

Update minimal-json dependencies

Minimal JSON library package has changed from: ""com.hazelcast.com.eclipsesource.json" (3.10) to "com.hazelcast.internal.json" (3.11). It needs to be updated.

NoSuchMethodError: io.fabric8.kubernetes.client.ConfigBuilder.build()Lio/fabric8/kubernetes/client/Config;

Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [io.fabric8.kubernetes.client.Config]: Factory method 'kubernetesClientConfig' threw exception; nested exception is java.lang.NoSuchMethodError: io.fabric8.kubernetes.client.ConfigBuilder.build()Lio/fabric8/kubernetes/client/Config;
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:189)
at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:588)
... 23 common frames omitted
Caused by: java.lang.NoSuchMethodError: io.fabric8.kubernetes.client.ConfigBuilder.build()Lio/fabric8/kubernetes/client/Config;

issues with kubernetes and preparing blog post with a sample

see the discussion here: https://gitter.im/hazelcast/hazelcast?at=5ae018a8109bb04332fb0f0d

https://blog.hazelcast.com/deploy-monitor/ could be outdated so we need to update it or just need to write a new one along with a code-samples...

Openshift example can be reused here.
https://github.com/hazelcast/hazelcast-code-samples/tree/master/hazelcast-integration/openshift

Typo in sample code: incorrect class name

I just want to report a typo on the main Github page of this plugin that caused me some headaches getting this plugin working.

In the DNS Lookup section of README.adoc, the discovery-strategy class name is incorrectly set to com.hazelcast.HazelcastKubernetesDiscoveryStrategy, where it should be set to com.hazelcast.kubernetes.HazelcastKubernetesDiscoveryStrategy instead.

Hazelcast doesn't throw any Exceptions or logs any errors whatsoever, which makes this error extremely annoying to realize.

NPE when no ready pods reported in Endpoints

Note

Potentially duplicate of #29

Scenario

When an HZ instance boots up and the ServiceEndpointResolver kicks in with a provided serviceName the retrieved Endpoints object via fabric8 Kubernetes-Client is iterated over endpoints.getSubsets() and endpointSubset.getAddresses(). But the latter is null because there are no ready endpoints. Raw json response:

{
  "kind": "Endpoints",
  "apiVersion": "v1",
  "metadata": {
    "name": "demo-service-9aadd30",
    "namespace": "dev",
    "selfLink": "/api/v1/namespaces/dev/endpoints/demo-service-9aadd30",
    "uid": "8a4fcf99-92ea-11e7-846d-005056b85198",
    "resourceVersion": "10935908",
    "creationTimestamp": "2017-09-06T10:02:49Z",
    "labels": {
      "appName": "demo-service",
      "chart": "demo-service-1.0.7-SNAPSHOT",
      "release": "demo-service-9aadd30"
    }
  },
  "subsets": [
    {
      "notReadyAddresses": [
        {
          "ip": "172.16.1.60",
          "nodeName": "node03",
          "targetRef": {
            "kind": "Pod",
            "namespace": "dev",
            "name": "demo-service-9aadd30-3817585861-dd1zz",
            "uid": "4abc9584-92f5-11e7-846d-005056b85198",
            "resourceVersion": "10935516"
          }
        },
        {
          "ip": "172.16.1.61",
          "nodeName": "comp03",
          "targetRef": {
            "kind": "Pod",
            "namespace": "dev",
            "name": "demo-service-9aadd30-test",
            "uid": "c4f0c539-9302-11e7-bbe2-005056b89983",
            "resourceVersion": "10935907"
          }
        },
        {
          "ip": "172.16.3.56",
          "nodeName": "comp05",
          "targetRef": {
            "kind": "Pod",
            "namespace": "dev",
            "name": "demo-service-9aadd30-3817585861-w56dd",
            "uid": "4ab4c940-92f5-11e7-846d-005056b85198",
            "resourceVersion": "10935514"
          }
        },
        {
          "ip": "172.16.8.53",
          "nodeName": "comp06",
          "targetRef": {
            "kind": "Pod",
            "namespace": "dev",
            "name": "demo-service-9aadd30-2520348926-b9d5p",
            "uid": "d0ef3797-92f4-11e7-846d-005056b85198",
            "resourceVersion": "10935702"
          }
        }
      ],
      "ports": [
        {
          "name": "service-port",
          "port": 8080,
          "protocol": "TCP"
        },
        {
          "name": "hazelcast-port",
          "port": 5701,
          "protocol": "TCP"
        }
      ]
    }
  ]
}

The same request but with a fake bootstrap-pod with matching selectors yields:


{
  "kind": "Endpoints",
  "apiVersion": "v1",
  "metadata": {
    "name": "demo-service-9aadd30",
    "namespace": "dev",
    "selfLink": "/api/v1/namespaces/dev/endpoints/demo-service-9aadd30",
    "uid": "8a4fcf99-92ea-11e7-846d-005056b85198",
    "resourceVersion": "10936092",
    "creationTimestamp": "2017-09-06T10:02:49Z",
    "labels": {
      "appName": "demo-service",
      "chart": "demo-service-1.0.7-SNAPSHOT",
      "release": "demo-service-9aadd30"
    }
  },
  "subsets": [
    {
      "addresses": [
        {
          "ip": "172.16.1.61",
          "nodeName": "comp03",
          "targetRef": {
            "kind": "Pod",
            "namespace": "dev",
            "name": "demo-service-9aadd30-test",
            "uid": "c4f0c539-9302-11e7-bbe2-005056b89983",
            "resourceVersion": "10936091"
          }
        }
      ],
      "notReadyAddresses": [
        {
          "ip": "172.16.1.60",
          "nodeName": "comp03",
          "targetRef": {
            "kind": "Pod",
            "namespace": "dev",
            "name": "demo-service-9aadd30-3817585861-dd1zz",
            "uid": "4abc9584-92f5-11e7-846d-005056b85198",
            "resourceVersion": "10935516"
          }
        },
        {
          "ip": "172.16.3.56",
          "nodeName": "comp05",
          "targetRef": {
            "kind": "Pod",
            "namespace": "dev",
            "name": "demo-service-9aadd30-3817585861-w56dd",
            "uid": "4ab4c940-92f5-11e7-846d-005056b85198",
            "resourceVersion": "10935514"
          }
        },
        {
          "ip": "172.16.8.53",
          "nodeName": "comp06",
          "targetRef": {
            "kind": "Pod",
            "namespace": "dev",
            "name": "demo-service-9aadd30-2520348926-b9d5p",
            "uid": "d0ef3797-92f4-11e7-846d-005056b85198",
            "resourceVersion": "10935702"
          }
        }
      ],
      "ports": [
        {
          "name": "service-port",
          "port": 8080,
          "protocol": "TCP"
        },
        {
          "name": "hazelcast-port",
          "port": 5701,
          "protocol": "TCP"
        }
      ]
    }
  ]
}

And the deployment is successfull.

Culprint

Internal fabric8 uses Jackson to unmarshel json to Endpoints an non-existing addresses JSON-objects results to a null value for the addresses field. Hence

hazelcast-kubernetes/src/main/java/com/hazelcast/kubernetes/ServiceEndpointResolver.java

Line 116 in 4884098

for (EndpointAddress endpointAddress : endpointSubset.getAddresses()) {

produces a NullPointerException.

Possible solution

Add a null check on each subsets loop. e.g. :

if (endpointSubset.getAddresses() == null) {
       continue;
 }

Please sign the CLA (Contributors License Agreement)

Hey guys,

We would like to go forward and officially support the kubernetes plugin. That includes moving the plugin repository to the Hazelcast GitHub account and handing over responsibility to our integrations team.

To make this step it would be great if you could help out and sign our Contributors License Agreement to give us full permissions over the contributions you did. I really appreciate your help and time and hope to see more contributions in the future :)

CLA necessary from:

@jimmidyson *
@amoAHCP *
@ReToCode *
@yurybubnov
@mrahbar (since I would like to merge your PR as well) *

You can find information as well as the instructions on our public part of the wiki: https://hazelcast.atlassian.net/wiki/display/COM/Hazelcast+Contributor+Agreement

Thanks again and I hope to see further PRs coming along :)

* already signed

User "system:serviceaccount:poc:default" cannot get endpoints in project

User "system:serviceaccount:poc:default" cannot get endpoints in project ****, do we need to create a new sa for deploying application?

Switch to SRV lookups for service endpoints

Currently connections will be made to the service IP rather than the pod IPs. Switching to SRV records will enable discovery of pod IPs & ports from DNS.

I'll submit a PR to see how this looks & what you think.

Property names in the README are not correct

They need to be prefixed with:

hazelcast.kubernetes. in order to work

How to make non-ready pods discoverable by the plugin?

If two pods are initialized in parallel they don't see each other (because the Readiness Probe returns false) and form two separate clusters. Is there any way to prevent this behavior?

Add retries for node discovery

The Kubernetes REST API should be called with retries. Otherwise, there is a risk of a sudden failure just because Kubernetes Master is down.

SocketException[Connection refused to address /172.16.60.55:5701]

Hello, I am having issues while trying to have hazelcast connecting to other nodes. Let me know if I am missing anything. I would assume that if should find other nodes and auto connect to it.

Here is my k8 deployment:

NAME: hazelcast
REVISION: 1
RELEASED: Wed Feb 28 13:46:56 2018
CHART: hazelcast-testhz-0.1.0
USER-SUPPLIED VALUES:
env:
gitCommit: f34wew2a5aer33ffas7a
springProfile: qa
image:
pullPolicy: Always
repository: hazelcast/hazelcast-kubernetes
tag: 3.10-BETA-1
ingress:
enabled: true
hosts:

hazelcast.us-west-2.nonprod.cnqr.delivery
tls:
hosts:
- hazelcast.us-west-2.nonprod.cnqr.delivery
  nginx:
  configmap:
  enableunderscore: true
  pod:
  initialDelaySeconds: 120
  timeoutSeconds: 60
  replicaCount: 3
  resources:
  limits:
  cpu: 2
  memory: 4Gi
  requests:
  cpu: 600m
  memory: 256Mi
  service:
  annotations:
  service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
  service.beta.kubernetes.io/aws-load-balancer-internal: 0.0.0.0/0
  externalPort: 5071
  internalPort: 5071
  name: hazelcast
  namespace: testhz
  type: ClusterIP

COMPUTED VALUES:
env:
gitCommit: f34wew2a5aer33ffas7a
springProfile: qa
image:
pullPolicy: Always
repository: hazelcast/hazelcast-kubernetes
tag: 3.10-BETA-1
ingress:
enabled: true
hosts:

hazelcast.us-west-2.nonprod.cnqr.delivery
tls:
hosts:
- hazelcast.us-west-2.nonprod.cnqr.delivery
  nginx:
  configmap:
  enableunderscore: true
  pod:
  initialDelaySeconds: 120
  timeoutSeconds: 60
  replicaCount: 3
  resources:
  limits:
  cpu: 2
  memory: 4Gi
  requests:
  cpu: 600m
  memory: 256Mi
  service:
  annotations:
  service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
  service.beta.kubernetes.io/aws-load-balancer-internal: 0.0.0.0/0
  externalPort: 5071
  internalPort: 5071
  name: hazelcast
  namespace: testhz
  type: ClusterIP

HOOKS:
MANIFEST:

Source: hazelcast-testhz/templates/nginx-configmap.yaml

apiVersion: v1
kind: ConfigMap
metadata:
name: hz-config
namespace: testhz
data:
HAZELCAST_KUBERNETES_NAMESPACE: testhz
HAZELCAST_KUBERNETES_SERVICE_NAME: hazelcast

Source: hazelcast-testhz/templates/service.yaml

apiVersion: v1
kind: Service
metadata:
name: hazelcast
labels:
app: hazelcast
spec:
type: NodePort
selector:
app: hazelcast
ports:

protocol: TCP
port: 5701
name: hzport

Source: hazelcast-testhz/templates/deployment.yaml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: hazelcast
namespace: testhz
labels:
app: hazelcast
chart: hazelcast-0.1.0
release: hazelcast-hazelcast
heritage: Tiller-hazelcast
spec:
replicas: 3
template:
metadata:
labels:
app: hazelcast
release: hazelcast-hazelcast
spec:
containers:
- name: hazelcast
image: "hazelcast/hazelcast-kubernetes:3.10-BETA-1"
imagePullPolicy: Always
ports:
- containerPort: 5071
livenessProbe:
exec:
command:
- "./liveness.sh"
initialDelaySeconds: 120
timeoutSeconds: 60
periodSeconds: 10
resources:
limits:
cpu: 2
memory: 4Gi
requests:
cpu: 600m
memory: 256Mi

Source: hazelcast-testhz/templates/ingress.yaml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: hazelcast
namespace: testhz
labels:
app: hazelcast
chart: hazelcast-0.1.0
release: hazelcast-hazelcast
heritage: Tiller
annotations:
spec:
rules:
- host: hazelcast.us-west-2.nonprod.cnqr.delivery
http:
paths:
- path: /
backend:
serviceName: hazelcast
servicePort: 5071
tls:
- hosts:
- hazelcast.us-west-2.nonprod.cnqr.delivery
LAST DEPLOYED: Wed Feb 28 13:46:56 2018
NAMESPACE: testhz
STATUS: DEPLOYED

RESOURCES:
==> v1/Service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
hazelcast NodePort 192.168.143.243 5701:31195/TCP 1s

==> v1beta1/Deployment
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
hazelcast 3 3 3 0 1s

Kube logs

╰─ kcwizlog po/hazelcast-2366664114-202vc
Kubernetes Namespace: testhz
Kubernetes Service DNS:
########################################

RUN_JAVA=

JAVA_OPTS=

CLASSPATH=/:/opt/hazelcast/:/opt/hazelcast/external/*:

########################################
Checking custom configuration
no custom configuration found
Feb 28, 2018 9:47:15 PM com.hazelcast.config.XmlConfigLocator
INFO: Loading 'hazelcast.xml' from working directory.
Feb 28, 2018 9:47:15 PM com.hazelcast.config.AbstractXmlConfigHelper
WARNING: Name of the hazelcast schema location is incorrect, using default
Feb 28, 2018 9:47:16 PM com.hazelcast.instance.AddressPicker
INFO: [LOCAL] [dev] [3.10-BETA-1] Prefer IPv4 stack is true.
Feb 28, 2018 9:47:16 PM com.hazelcast.instance.AddressPicker
INFO: [LOCAL] [dev] [3.10-BETA-1] Picked [172.16.18.169]:5701, using socket ServerSocket[addr=/0.0.0.0,localport=5701], bind any local is true
Feb 28, 2018 9:47:16 PM com.hazelcast.system
INFO: [172.16.18.169]:5701 [dev] [3.10-BETA-1] Hazelcast 3.10-BETA-1 (20180221 - c9a445b) starting at [172.16.18.169]:5701
Feb 28, 2018 9:47:16 PM com.hazelcast.system
INFO: [172.16.18.169]:5701 [dev] [3.10-BETA-1] Copyright (c) 2008-2018, Hazelcast, Inc. All Rights Reserved.
Feb 28, 2018 9:47:16 PM com.hazelcast.system
INFO: [172.16.18.169]:5701 [dev] [3.10-BETA-1] Configured Hazelcast Serialization version: 1
Feb 28, 2018 9:47:16 PM com.hazelcast.spi.impl.operationservice.impl.BackpressureRegulator
INFO: [172.16.18.169]:5701 [dev] [3.10-BETA-1] Backpressure is disabled
Feb 28, 2018 9:47:16 PM com.hazelcast.spi.discovery.integration.DiscoveryService
INFO: [172.16.18.169]:5701 [dev] [3.10-BETA-1] Kubernetes Discovery properties: { service-dns: null, service-dns-timeout: 5, service-name: null, service-label: null, service-label-value: true, namespace: default, resolve-not-ready-addresses: false, kubernetes-master: https://kubernetes.default.svc}
Feb 28, 2018 9:47:16 PM com.hazelcast.spi.discovery.integration.DiscoveryService
INFO: [172.16.18.169]:5701 [dev] [3.10-BETA-1] Kubernetes Discovery: Bearer Token { eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJ3aXphcmRzIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImRlZmF1bHQtdG9rZW4tbWx6NGoiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGVmYXVsdCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6Ijk0MDNlNGJiLWMwZWEtMTFlNy05NTk4LTBhMGIxNzMwNDgwMiIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDp3aXphcmRzOmRlZmF1bHQifQ.WBNhlfnXJ6_1d3KfVitj6L_tHXizIn-OgOpZV6DbZjataKWY37VlJBToSdJTvODlOh6V9LvI7C7bQl66DEpuhr3netv7Z0AFPFk1nlOvgblVwRDUqDNI5tTayD3sr5oIVRNNOqljPPa89UIYwCkL8BKxmZwMIE97r_tFRWEW01psU8U68UgE-2nudMHBkyRtRlKlRJsMkU79N9Y2Fbd1DqsNrTtenVyO9ZeTwLMw7cuhwvHl6N4wDjerJDM8VV4yjv5YUkrFD9ID0GINt6lXHCeTxcESkKxfvYVUxCAEKs2ixyMXIskwhI_fNvzetVzMfH95Fnu153amuwrvVtpqIA }
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Feb 28, 2018 9:47:17 PM com.hazelcast.spi.discovery.integration.DiscoveryService
INFO: [172.16.18.169]:5701 [dev] [3.10-BETA-1] Kubernetes Discovery activated resolver: ServiceEndpointResolver
Feb 28, 2018 9:47:17 PM com.hazelcast.instance.Node
INFO: [172.16.18.169]:5701 [dev] [3.10-BETA-1] Activating Discovery SPI Joiner
Feb 28, 2018 9:47:17 PM com.hazelcast.spi.impl.operationexecutor.impl.OperationExecutorImpl
INFO: [172.16.18.169]:5701 [dev] [3.10-BETA-1] Starting 16 partition threads and 9 generic threads (1 dedicated for priority tasks)
Feb 28, 2018 9:47:17 PM com.hazelcast.internal.diagnostics.Diagnostics
INFO: [172.16.18.169]:5701 [dev] [3.10-BETA-1] Diagnostics disabled. To enable add -Dhazelcast.diagnostics.enabled=true to the JVM arguments.
Feb 28, 2018 9:47:17 PM com.hazelcast.core.LifecycleService
INFO: [172.16.18.169]:5701 [dev] [3.10-BETA-1] [172.16.18.169]:5701 is STARTING
Feb 28, 2018 9:47:18 PM com.hazelcast.nio.tcp.TcpIpConnector
INFO: [172.16.18.169]:5701 [dev] [3.10-BETA-1] Connecting to /10.253.67.76:5701, timeout: 0, bind-any: true
Feb 28, 2018 9:47:18 PM com.hazelcast.nio.tcp.TcpIpConnector
INFO: [172.16.18.169]:5701 [dev] [3.10-BETA-1] Connecting to /172.16.17.122:5701, timeout: 0, bind-any: true
Feb 28, 2018 9:47:18 PM com.hazelcast.nio.tcp.TcpIpConnector
INFO: [172.16.18.169]:5701 [dev] [3.10-BETA-1] Connecting to /172.16.13.23:5701, timeout: 0, bind-any: true
Feb 28, 2018 9:47:18 PM com.hazelcast.nio.tcp.TcpIpConnector
INFO: [172.16.18.169]:5701 [dev] [3.10-BETA-1] Connecting to /172.16.60.55:5701, timeout: 0, bind-any: true
Feb 28, 2018 9:47:18 PM com.hazelcast.nio.tcp.TcpIpConnector
INFO: [172.16.18.169]:5701 [dev] [3.10-BETA-1] Could not connect to: /172.16.17.122:5701. Reason: SocketException[Connection refused to address /172.16.17.122:5701]
Feb 28, 2018 9:47:18 PM com.hazelcast.nio.tcp.TcpIpConnector
INFO: [172.16.18.169]:5701 [dev] [3.10-BETA-1] Could not connect to: /172.16.13.23:5701. Reason: SocketException[Connection refused to address /172.16.13.23:5701]
Feb 28, 2018 9:47:18 PM com.hazelcast.nio.tcp.TcpIpConnector
INFO: [172.16.18.169]:5701 [dev] [3.10-BETA-1] Could not connect to: /172.16.60.55:5701. Reason: SocketException[Connection refused to address /172.16.60.55:5701]
Feb 28, 2018 9:47:18 PM com.hazelcast.internal.cluster.impl.DiscoveryJoiner
INFO: [172.16.18.169]:5701 [dev] [3.10-BETA-1] [172.16.17.122]:5701 is added to the blacklist.
Feb 28, 2018 9:47:18 PM com.hazelcast.internal.cluster.impl.DiscoveryJoiner
INFO: [172.16.18.169]:5701 [dev] [3.10-BETA-1] [172.16.60.55]:5701 is added to the blacklist.
Feb 28, 2018 9:47:18 PM com.hazelcast.internal.cluster.impl.DiscoveryJoiner
INFO: [172.16.18.169]:5701 [dev] [3.10-BETA-1] [172.16.13.23]:5701 is added to the blacklist.
Feb 28, 2018 9:47:23 PM com.hazelcast.system
INFO: [172.16.18.169]:5701 [dev] [3.10-BETA-1] Cluster version set to 3.10
Feb 28, 2018 9:47:23 PM com.hazelcast.internal.cluster.ClusterService
INFO: [172.16.18.169]:5701 [dev] [3.10-BETA-1]

Members {size:1, ver:1} [
Member [172.16.18.169]:5701 - 1787f7b3-afbb-4dab-a375-c9d55f5ecbcf this
]

Feb 28, 2018 9:47:23 PM com.hazelcast.core.LifecycleService
INFO: [172.16.18.169]:5701 [dev] [3.10-BETA-1] [172.16.18.169]:5701 is STARTED

Could hazlecast-kuberntes work with 3.6.5?

Hi, sorry for the question posted here.
I'm working to deploy OrientDB 2.2.x on Google Containers.
ODB uses HZ 3.6.5, so I recompiled this project for it without any error in the build.
At runtime the com.hazelcast.HazelcastKubernetesDiscoveryStrategy isn't loaded.
Maybe I need to modify my configuration file or to modify the code to let it work with HZ 3.6.5

thank you in advance for the support

Token retrieval from file not working

when trying to use the REST API method, it fails to read the token from file, however also doesnt throw an exception that reading fails.

all you see is a null for the token

INFO: [10.4.1.222]:5701 [dev] [3.7.4] Kubernetes Discovery properties: { service-dns: null, service-dns-timeout: 5, service-name: null, service-label: vertx-cluster, service-label-value: true, namespace: production, kubernetes-master: https://kubernetes.default.svc}
Sep 06, 2017 1:44:03 PM com.hazelcast.spi.discovery.integration.DiscoveryService
INFO: [10.4.1.222]:5701 [dev] [3.7.4] Kubernetes Discovery: Bearer Token { null }

and later

SEVERE: [10.4.1.222]:5701 [dev] [3.7.4] null
java.lang.NullPointerException
    at com.hazelcast.kubernetes.ServiceEndpointResolver.getSimpleDiscoveryNodes(ServiceEndpointResolver.java:115)
    at com.hazelcast.kubernetes.ServiceEndpointResolver.getDiscoveryNodes(ServiceEndpointResolver.java:105)
    at com.hazelcast.kubernetes.ServiceEndpointResolver.resolve(ServiceEndpointResolver.java:84)
    at com.hazelcast.kubernetes.HazelcastKubernetesDiscoveryStrategy.discoverNodes(HazelcastKubernetesDiscoveryStrategy.java:98)

if you specify the token via property api-token , its all working fine

PS: DNS method didnt work at all but no errors

Update kubernetes-client version

HI,
when using vertx application on openshift (vertx uses hazelcast as cluster manager) I get a conflict with the kubernetes-client version as this hazelcast-kubernetes project still uses old version 1.3.66 while the vertx-config-kubernetes-configmap uses version 1.4.35. Having both installed the configmap module doesnt work.

Instruction doesn't applied for Hazelcast 3.9.1

For the Hazelcast 3.9.1

I've tried with with configuration

<!-- activate the Kubernetes plugin -->
      <discovery-strategies>
        <discovery-strategy enabled="true"
            class="com.hazelcast.kubernetes.HazelcastKubernetesDiscoveryStrategy">

          <properties>
            <!-- configure discovery service API lookup -->
            <property name="service-name">MY-SERVICE-NAME</property>
             <property name="service-label-name">cluster01</property>
              <property name="service-label-value">true</property>
            <property name="namespace">MY-KUBERNETES-NAMESPACE</property>
          </properties>
        </discovery-strategy>
      </discovery-strategies>

But I then encounter an error shown as below:

Caused by: com.hazelcast.config.properties.ValidationException: There is no discovery strategy factory to create 'DiscoveryStrategyConfig{properties={namespace=default}, className='com.hazelcast.kubernetes.HazelcastKubernetesDiscoveryStrategy', discoveryStrategyFactory=null}' Is it a typo in a strategy classname? Perhaps you forgot to include implementation on a classpath?
        at com.hazelcast.spi.discovery.impl.DefaultDiscoveryService.buildDiscoveryStrategy(DefaultDiscoveryService.java:188)
        at com.hazelcast.spi.discovery.impl.DefaultDiscoveryService.loadDiscoveryStrategies(DefaultDiscoveryService.java:145)

Is it neccesary to configure discovery on client side when using a Kubernetes service?

If clients connect to server through a Kubernetes service, Is it neccesary to configure discovery on client side?

PROPERTY_DEFINITIONS is missing SERVICE_DNS_TIMEOUT in HazelcastKubernetesDiscoveryStrategyFactory.java

Right after this line is where the property should be declared...

Incorrect PORT from SRV record

I am running a kubernetes cluster with skyDNS service on

The hazelcast container logs indicate that it is trying to connect on port :0

Nov 15, 2016 10:35:11 PM com.hazelcast.nio.tcp.InitConnectionTask INFO: [11.11.104.3]:5701 [dev] [3.6.1] Connecting to /11.11.109.2:0, timeout: 0, bind-any: true Nov 15, 2016 10:35:11 PM com.hazelcast.nio.tcp.InitConnectionTask INFO: [11.11.104.3]:5701 [dev] [3.6.1] Connecting to /11.11.104.3:0, timeout: 0, bind-any: true Nov 15, 2016 10:35:11 PM com.hazelcast.nio.tcp.InitConnectionTask INFO: [11.11.104.3]:5701 [dev] [3.6.1] Could not connect to: /11.11.104.3:0. Reason: SocketException[Connection refused to address /11.11.104.3:0] Nov 15, 2016 10:35:11 PM com.hazelcast.cluster.impl.DiscoveryJoiner INFO: [11.11.104.3]:5701 [dev] [3.6.1] Address[11.11.104.3]:0 is added to the blacklist. Nov 15, 2016 10:35:11 PM com.hazelcast.nio.tcp.InitConnectionTask INFO: [11.11.104.3]:5701 [dev] [3.6.1] Could not connect to: /11.11.109.2:0. Reason: SocketException[Connection refused to address /11.11.109.2:0] Nov 15, 2016 10:35:11 PM com.hazelcast.cluster.impl.DiscoveryJoiner INFO: [11.11.104.3]:5701 [dev] [3.6.1] Address[11.11.109.2]:0 is added to the blacklist. Nov 15, 2016 10:35:12 PM com.hazelcast.cluster.impl.DiscoveryJoiner

I have created a headless kubernetes service for hazelcast and nslookup returns
ip addresses of all 3 hazelcast pods

nslookup hazelcast.kube-infra.svc.cluster.local 172.16.48.2
Server: 172.16.48.2
Address 1: 172.16.48.2 kube-dns.kube-system.svc.cluster.local

Name: hazelcast.kube-infra.svc.cluster.local
Address 1: 11.11.38.2
Address 2: 11.11.109.2
Address 3: 11.11.104.3

I am wondering how to make it connect to 5701 (which is the port where hazelcast is running in all pods)

kubectl describe svc hazelcast --namespace=kube-infra
Name: hazelcast
Namespace: kube-infra
Labels: app=hazelcast
version=latest
Selector: app=hazelcast,version=latest
Type: ClusterIP
IP: None
Port: tcp-5701-5701-nfj29 5701/TCP
Endpoints: 11.11.104.3:5701,11.11.109.2:5701,11.11.38.2:5701
Session Affinity: None

HZ not joining cluster in kubernetes

Hello,

I have built a k8s pod that contains HZ 3.7.2 with hazelcast-kubernetes-discovery-0.9.3.jar and dnsjava-2.1.7.jar on the classpath and have configured hazelcast with

<discovery-strategies>
  <discovery-strategy enabled="true"    class="com.noctarius.hazelcast.kubernetes.HazelcastKubernetesDiscoveryStrategy">
    <properties>
      <!--k8s service name-->
      <property name="service-dns">hazelcast</property>
     </properties>
    </discovery-strategy>
</discovery-strategies>

When starting up two pods in my deployment, I see in both logs:

> 2016-11-14T15:44:52.629474000Z ########################################
2016-11-14T15:44:52.629621000Z # RUN_JAVA=
2016-11-14T15:44:52.629762000Z # JAVA_OPTS=-Dhazelcast.rest.enabled=true
2016-11-14T15:44:52.629918000Z # starting now....
2016-11-14T15:44:52.630059000Z ########################################
2016-11-14T15:44:52.630204000Z Process id for hazelcast instance is written to location:  /opt/hazelcast/hazelcast_instance.pid
2016-11-14T15:44:52.920870000Z Nov 14, 2016 3:44:52 PM com.hazelcast.config.XmlConfigLocator
2016-11-14T15:44:52.921057000Z INFO: Loading 'hazelcast.xml' from working directory.
2016-11-14T15:44:53.330555000Z Nov 14, 2016 3:44:53 PM com.hazelcast.instance.DefaultAddressPicker
2016-11-14T15:44:53.330736000Z INFO: [LOCAL] [hlx] [3.7.2] Prefer IPv4 stack is true.
2016-11-14T15:44:53.355237000Z Nov 14, 2016 3:44:53 PM com.hazelcast.instance.DefaultAddressPicker
2016-11-14T15:44:53.355397000Z INFO: [LOCAL] [hlx] [3.7.2] Picked [10.36.0.5]:5701, using socket ServerSocket[addr=/0:0:0:0:0:0:0:0,localport=5701], bind any local is true
2016-11-14T15:44:53.375015000Z Nov 14, 2016 3:44:53 PM com.hazelcast.system
2016-11-14T15:44:53.375212000Z INFO: [10.36.0.5]:5701 [hlx] [3.7.2] Hazelcast 3.7.2 (20161004 - 540b01c) starting at [10.36.0.5]:5701
2016-11-14T15:44:53.375965000Z Nov 14, 2016 3:44:53 PM com.hazelcast.system
2016-11-14T15:44:53.376126000Z INFO: [10.36.0.5]:5701 [hlx] [3.7.2] Copyright (c) 2008-2016, Hazelcast, Inc. All Rights Reserved.
2016-11-14T15:44:53.376977000Z Nov 14, 2016 3:44:53 PM com.hazelcast.system
2016-11-14T15:44:53.377147000Z INFO: [10.36.0.5]:5701 [hlx] [3.7.2] Configured Hazelcast Serialization version : 1
2016-11-14T15:44:53.562107000Z Nov 14, 2016 3:44:53 PM com.hazelcast.spi.impl.operationservice.impl.BackpressureRegulator
2016-11-14T15:44:53.562288000Z INFO: [10.36.0.5]:5701 [hlx] [3.7.2] Backpressure is disabled
2016-11-14T15:44:53.994721000Z Nov 14, 2016 3:44:53 PM com.hazelcast.spi.discovery.integration.DiscoveryService
2016-11-14T15:44:53.994916000Z INFO: [10.36.0.5]:5701 [hlx] [3.7.2] Kubernetes Discovery properties: { service-dns: hazelcast, service-name: null, service-label: null, service-label-value: true, namespace: default}
2016-11-14T15:44:53.996166000Z Nov 14, 2016 3:44:53 PM com.hazelcast.spi.discovery.integration.DiscoveryService
2016-11-14T15:44:53.996319000Z INFO: [10.36.0.5]:5701 [hlx] [3.7.2] Kubernetes Discovery activated resolver: DnsEndpointResolver
2016-11-14T15:44:53.997551000Z Nov 14, 2016 3:44:53 PM com.hazelcast.core.LifecycleService
2016-11-14T15:44:53.997700000Z INFO: [10.36.0.5]:5701 [hlx] [3.7.2] [10.36.0.5]:5701 is STARTING
2016-11-14T15:44:54.117326000Z Nov 14, 2016 3:44:54 PM com.hazelcast.spi.impl.operationexecutor.impl.OperationExecutorImpl
2016-11-14T15:44:54.117504000Z INFO: [10.36.0.5]:5701 [hlx] [3.7.2] Starting 2 partition threads
2016-11-14T15:44:54.120001000Z Nov 14, 2016 3:44:54 PM com.hazelcast.spi.impl.operationexecutor.impl.OperationExecutorImpl
2016-11-14T15:44:54.120160000Z INFO: [10.36.0.5]:5701 [hlx] [3.7.2] Starting 3 generic threads (1 dedicated for priority tasks)
2016-11-14T15:44:54.127213000Z Nov 14, 2016 3:44:54 PM com.hazelcast.nio.tcp.nonblocking.NonBlockingIOThreadingModel
2016-11-14T15:44:54.127368000Z INFO: [10.36.0.5]:5701 [hlx] [3.7.2] TcpIpConnectionManager configured with Non Blocking IO-threading model: 3 input threads and 3 output threads
2016-11-14T15:44:54.141210000Z Nov 14, 2016 3:44:54 PM com.hazelcast.instance.Node
2016-11-14T15:44:54.141367000Z WARNING: [10.36.0.5]:5701 [hlx] [3.7.2] No join method is enabled! Starting standalone.
2016-11-14T15:44:54.168042000Z Nov 14, 2016 3:44:54 PM com.hazelcast.core.LifecycleService
2016-11-14T15:44:54.168217000Z INFO: [10.36.0.5]:5701 [hlx] [3.7.2] [10.36.0.5]:5701 is STARTED

So far, so good. One pod is on 10.36.0.5, the other on 10.32.0.6. If I shell into a pod in the cluster and do an nslookup on the service name 'hazelcast', I get both addresses:

> nslookup hazelcast
Name:      hazelcast
Address 1: 10.36.0.5
Address 2: 10.32.0.6

But when I query the cluster status from either of the hazelcast pods, I see that they are not joined:

curl 10.32.0.6:5701/hazelcast/rest/cluster
Members [1] {
Member [10.32.0.6]:5701 - f0afb098-55d7-428c-bc95-768760029590 this
}

curl 10.36.0.5:5701/hazelcast/rest/cluster
Members [1] {
Member [10.36.0.5]:5701 - dbba32b7-353d-400e-820c-1ce95d29079d this
}

So the HZ instances are starting with dns lookup enabled, but are not joining. What am I doing wrong?
regards
Peter

endpoints.getSubsets() should be checked for null

endpoints.getSubsets() in getSimpleDiscoveryNodes may return null in some circumstances.
As we don't check for null, this would result in an NPE being thrown.
See #42 (comment)

Automatic discovery of all nodes in namespace causes trouble with 2 or more separate HZ clusters

Hi,

with 0.9.3 version I saw that resolve() was made "smarter" (https://github.com/noctarius/hazelcast-kubernetes-discovery/blob/master/src/main/java/com/noctarius/hazelcast/kubernetes/ServiceEndpointResolver.java#L68). But in my case that is quite a problem.

We have two separate hazelcast clusters on kubernetes (in same namespace). Let's say I start one of them. Two pods come up and I'd expect they will eventually form a cluster (but until hz is intialized readiness check prevents one to see the other node). That means I expect the discovery plugin to say there are no other nodes. What happens instead is that nodes from the other cluster are discovered but credentials are incorrect so at the end hz can't connect anywhere and just terminates.

2016-11-11 13:40:13,247 [localhost-startStop-1] [DEBUG] [c.h.i.c.impl.DiscoveryJoiner] - [10.201.36.12]:5701 [mts-dev-ts-cluster] [3.7.2] PostJoin master: null, isMaster: false
2016-11-11 13:40:13,248 [localhost-startStop-1] [ERROR] [com.hazelcast.instance.Node] - [10.201.36.12]:5701 [mts-dev-ts-cluster] [3.7.2] Could not join cluster in 300000 ms. Shutting down now!
2016-11-11 13:40:13,250 [localhost-startStop-1] [INFO ] [c.h.core.LifecycleService] - [10.201.36.12]:5701 [mts-dev-ts-cluster] [3.7.2] [10.201.36.12]:5701 is SHUTTING_DOWN
2016-11-11 13:40:13,251 [localhost-startStop-1] [WARN ] [com.hazelcast.instance.Node] - [10.201.36.12]:5701 [mts-dev-ts-cluster] [3.7.2] Terminating forcefully...

Can this additional logic to discover nodes be disabled somehow?

Javadoc discrepancy

Hi!
There is minor discrepancy in property definition (missed dash in resolve-not-ready-addresses) between the javadoc and the code:

hazelcast-kubernetes/src/main/java/com/hazelcast/kubernetes/KubernetesProperties.java

Line 89 in 833f232

* <p>Configuration key: <tt>resolve-notready-addresses</tt></p>

and

hazelcast-kubernetes/src/main/java/com/hazelcast/kubernetes/KubernetesProperties.java

Line 92 in 833f232

    
           public static final PropertyDefinition RESOLVE_NOT_READY_ADDRESSES = property("resolve-not-ready-addresses", BOOLEAN);

Make Kubernetes ZONE_AWARE

Use the label value of failure-domain.beta.kubernetes.io/zone to retrieve the zone and make Hazelcast Zone Aware.

The label can be retrieved as follows from Kubernetes Master:

Get nodeName from /api/v1/namespaces/<namespace>/endpoints
Get label value from /api/v1/nodes/<nodeName>

Note that the RBAC sample yaml (+ Helm Charts) need to be adapted to give read access to the /nodes endpoint.

Write Blog Post about using Kubernetes

Write a Blog Post or a series of blog posts about:

Using Kubernetes plugin
- with DNS Lookup Discovery
- with REST API Discovery
Using Hazelcast Helm Chart

Kubernetes [1] and OpenShift [2] code samples can be used.

[1] https://github.com/hazelcast/hazelcast-code-samples/tree/master/hazelcast-integration/kubernetes
[2] https://github.com/hazelcast/hazelcast-code-samples/tree/master/hazelcast-integration/openshift

fix document about the dns lookup to use headless service

As I test, this plugin only works with headless service name when using dns lookup. Please fix the document.

Thanks

New release?

Any chance of a release please?

Hazelcast port not configurable

When using the DnsEndpointResolver, the Hazelcast port cannot be configured: the DnsEndpointResolver.getHazelcastPort() method always returns the static DEFAULT_PORT value. Could this port value be configurable, just as it is when using the ServiceEndpointResolver?

Integration with kubernetes horizontal pod autoscaling

This is a feature request to integrate Hazelcast with Kubernete's horizontal pod autoscaling (HPA).

The newer autoscaling/v2beta1 version of HPA allows for memory and custom metrics to be used instead of CPU utilization, however, some experiments with this API illustrated that HZ pods could scale up but did not scale down (i.e. they did not trigger a scale down threshold).

This feature request is for an investigation, design and implementation on how to best integrate HZ's metrics with kubernetes such that both scaling up and down occurs.

"Wrong bind request from..." message on Hazelcast startup in Google's Kubernetes

First, thanks so much for writing and maintaining this project, it's exactly what I need.

So I've written a simple Java app that uses a Hazelcast Lock to ensure that a task only runs on one instance (the others bale when the lock is held). The app sits in a permanent loop attempting to run the task every 2 seconds and the task itself takes 5 seconds, so I can see plenty of lock contention.

This works fine on my local computer so I've now packaged it up as an image and am attempting to run it on Google's Container Engine.

I'm configuring Hazelcast through Spring like so:

<hz:hazelcast id="instance">
		<hz:config>
			<hz:properties>
				<hz:property name="hazelcast.discovery.enabled">true</hz:property>
			</hz:properties>
			<hz:network port="5701" port-auto-increment="true">
				<hz:join>
					<hz:multicast enabled="false" />
					<hz:tcp-ip enabled="false" />

					<hz:discovery-strategies>

						<hz:discovery-strategy
							class-name="com.noctarius.hazelcast.kubernetes.HazelcastKubernetesDiscoveryStrategy">
							<hz:properties>
								<hz:property name="service-dns">hazeltest-service-simple</hz:property>
								<hz:property name="service-dns-timeout">10</hz:property>
							</hz:properties>
						</hz:discovery-strategy>
					</hz:discovery-strategies>

				</hz:join>

			</hz:network>
		</hz:config>
	</hz:hazelcast>

The service definition I'm using looks like this:

apiVersion: v1
kind: Service
metadata:
  labels:
    name: hazeltest-name-label-simple
  name: hazeltest-service-simple
spec:
  ports:
    - port: 5701
      targetPort: 5701
      protocol: TCP
  selector:
    app: hazeltest-app-label

When I run the app, I can see the following in the logs:

2017-02-08T11:37:42.435702737Z Feb 08, 2017 11:37:42 AM com.hazelcast.nio.tcp.SocketAcceptorThread
2017-02-08T11:37:42.435758276Z INFO: [10.0.0.17]:5701 [dev] [3.7.5] Accepting socket connection from /10.0.1.12:52664
2017-02-08T11:37:42.461609384Z Feb 08, 2017 11:37:42 AM com.hazelcast.nio.tcp.TcpIpConnectionManager
2017-02-08T11:37:42.461679256Z INFO: [10.0.0.17]:5701 [dev] [3.7.5] Established socket connection between /10.0.0.17:5701 and /10.0.1.12:52664
2017-02-08T11:37:42.462780759Z Feb 08, 2017 11:37:42 AM com.hazelcast.nio.tcp.TcpIpConnectionManager
2017-02-08T11:37:42.462796237Z WARNING: [10.0.0.17]:5701 [dev] [3.7.5] Wrong bind request from [10.0.1.12]:5701! This node is not requested endpoint: [10.3.250.127]:5701
2017-02-08T11:37:42.464388597Z Feb 08, 2017 11:37:42 AM com.hazelcast.nio.tcp.TcpIpConnection
2017-02-08T11:37:42.464411912Z INFO: [10.0.0.17]:5701 [dev] [3.7.5] Connection[id=8, /10.0.0.17:5701->/10.0.1.12:52664, endpoint=null, alive=false, type=MEMBER] closed. Reason: Wrong bind request from [10.0.1.12]:5701! This node is not requested endpoint: [10.3.250.127]:5701

I'm running Hazelcast 3.7.5 and version 0.9.3 of this plugin, if that helps. Any pointers on why this would be happening or how to resolve would be super-appreciated.

Nodes/Instances are registered more than once

Hi,

i build a small spring boot app (https://github.com/codeCoder/spring-boot-hello-world)
to get involved with kubernetes and hazelcast in it.

During the start it looks like a single hazelcast instance is registered more than once in the cluster with different ports. In https://github.com/codeCoder/spring-boot-hello-world/blob/master/hello_world_last_run_first_pod
you see following lines

Members {size:3, ver:3} [
Member [172.17.0.8]:5701 - 96d9f638-9a22-4305-a68c-4e222b10e4aa this
Member [172.17.0.8]:5702 - 065b6ec2-3996-4230-a186-ebc4e5d6a3a5
Member [172.17.0.8]:5703 - b087dfc0-bd6b-4e9e-b0a7-44dd324e8034
]

If i start a second pod(see: https://github.com/codeCoder/spring-boot-hello-world/blob/master/hello_world_last_run_second_pod)

you see much more nodes/ instances registered in the cluster:

Members {size:6, ver:6} [
Member [172.17.0.8]:5701 - 96d9f638-9a22-4305-a68c-4e222b10e4aa
Member [172.17.0.8]:5702 - 065b6ec2-3996-4230-a186-ebc4e5d6a3a5
Member [172.17.0.8]:5703 - b087dfc0-bd6b-4e9e-b0a7-44dd324e8034
Member [172.17.0.11]:5701 - a8bc9171-e10f-4c26-9ad6-1a6fc43446ce
Member [172.17.0.11]:5702 - 1669f5a6-1fa5-442b-9745-c2e7a014be03 this
Member [172.17.0.11]:5703 - c41ac3df-f2d1-4ef7-88ca-8cf0afd78b54
]

At the moment I use this rest config: https://github.com/codeCoder/spring-boot-hello-world/blob/master/src/main/resources/hazelcast-kubernetes-restapi.xml

with following network settings:

    <port>5701</port>
    <join>
        <!-- deactivate normal discovery -->
        <multicast enabled="false"/>
        <tcp-ip enabled="false"/>
        <!-- activate the Kubernetes plugin -->
        <discovery-strategies>
            <discovery-strategy enabled="true"
                                class="com.hazelcast.kubernetes.HazelcastKubernetesDiscoveryStrategy">
                <properties>
                    <!-- configure discovery service API lookup -->
                    <property name="service-name">spring-boot-hello-world-service-cluster</property>
                    <property name="service-label-name">run</property>
                    <property name="service-label-value">spring-boot-hello-world-service-cluster</property>
                    <property name="namespace">default</property>
                </properties>
            </discovery-strategy>
        </discovery-strategies>
    </join>
</network>

I still have to test if the cache is working properly. But first I would like to understand this behavior.