Failure in HA upgrading from 15.1.1 to 16.0.1

[keycloak]

Failure in HA upgrading from 15.1.1 to 16.0.1

[stephane-dereppe]

Does anyone know any way to update the KeyCloak cluster without losing all the sessions from 15.1.1 to a higher KeyCloak version? Have you figured it out?

Not me. I just down scaled my cluster, upgrade and restart all nodes. (it was acceptable for my use cases..)

[julien-sarik]

What I understand from this issue is that Infinispan nodes from Keycloak 15 cannot coexist with the ones from Keycloak 16.
AFAIK keycloak does not support zero downtime upgrade yet.
It can be overcome by modifying the jgroups discovery policy.
For instance if you are using KUBE_PING protocol you can leverage Kubernetes labels to create a new Infinispan cluster along with the upgrade (see kube_ping discovery documentation).
That of course means all the data from the previous Infinispan cluster (e.g. the user sessions) will get lost with the upgrade.

[dejwsz]

I did such upgrade without any problems but to version 16.1.1 instead of 16.0.1. I had the same config you mentioned above regarding clustering. So maybe it is better to skip version 16.0.1.

[stephane-dereppe]

I got the same error upgrading from 15.1.1 to 16.1.1

[vilmosnagy]

Does anyone know any way to update the KeyCloak cluster without losing all the sessions from 15.1.1 to a higher KeyCloak version? Have you figured it out?

[weltonrodrigo]

Got the same kind of error upgrading from 17.0.1 to 18.0.2.

Workaround in kubernetes is to set replicas to 0 then set to previous scale again.

Log on a preexisting cluster member (17.0.1):

sso-keycloakx-0 keycloak 2022-07-03 17:27:03,869 ERROR [org.infinispan.CLUSTER] (jgroups-32,sso-keycloakx-0-46161) ISPN000474: Error processing request 5@sso-keycloakx-1-38522: java.io.EOFException
sso-keycloakx-0 keycloak 	at org.infinispan.marshall.core.AbstractBytesObjectInput.checkPosLength(AbstractBytesObjectInput.java:261)
sso-keycloakx-0 keycloak 	at org.infinispan.marshall.core.AbstractBytesObjectInput.readInt(AbstractBytesObjectInput.java:142)
sso-keycloakx-0 keycloak 	at org.infinispan.marshall.exts.ReplicableCommandExternalizer.readCommandParameters(ReplicableCommandExternalizer.java:119)
sso-keycloakx-0 keycloak 	at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.readObject(CacheRpcCommandExternalizer.java:164)
sso-keycloakx-0 keycloak 	at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.readObject(CacheRpcCommandExternalizer.java:86)
sso-keycloakx-0 keycloak 	at org.infinispan.marshall.core.GlobalMarshaller.readWithExternalizer(GlobalMarshaller.java:727)
sso-keycloakx-0 keycloak 	at org.infinispan.marshall.core.GlobalMarshaller.readNonNullableObject(GlobalMarshaller.java:708)
sso-keycloakx-0 keycloak 	at org.infinispan.marshall.core.GlobalMarshaller.readNullableObject(GlobalMarshaller.java:357)
sso-keycloakx-0 keycloak 	at org.infinispan.marshall.core.GlobalMarshaller.objectFromObjectInput(GlobalMarshaller.java:191)
sso-keycloakx-0 keycloak 	at org.infinispan.marshall.core.GlobalMarshaller.objectFromByteBuffer(GlobalMarshaller.java:220)
sso-keycloakx-0 keycloak 	at org.infinispan.remoting.transport.jgroups.JGroupsTransport.processRequest(JGroupsTransport.java:1440)
sso-keycloakx-0 keycloak 	at org.infinispan.remoting.transport.jgroups.JGroupsTransport.processMessage(JGroupsTransport.java:1380)
sso-keycloakx-0 keycloak 	at org.infinispan.remoting.transport.jgroups.JGroupsTransport.access$300(JGroupsTransport.java:138)
sso-keycloakx-0 keycloak 	at org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.up(JGroupsTransport.java:1524)
sso-keycloakx-0 keycloak 	at org.jgroups.JChannel.up(JChannel.java:780)
sso-keycloakx-0 keycloak 	at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:913)
sso-keycloakx-0 keycloak 	at org.jgroups.protocols.FRAG3.up(FRAG3.java:165)
sso-keycloakx-0 keycloak 	at org.jgroups.protocols.FlowControl.up(FlowControl.java:355)
sso-keycloakx-0 keycloak 	at org.jgroups.protocols.FlowControl.up(FlowControl.java:363)
sso-keycloakx-0 keycloak 	at org.jgroups.protocols.pbcast.GMS.up(GMS.java:876)
sso-keycloakx-0 keycloak 	at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:254)
sso-keycloakx-0 keycloak 	at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1048)
sso-keycloakx-0 keycloak 	at org.jgroups.protocols.UNICAST3.addMessage(UNICAST3.java:771)
sso-keycloakx-0 keycloak 	at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:752)
sso-keycloakx-0 keycloak 	at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:405)
sso-keycloakx-0 keycloak 	at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:592)
sso-keycloakx-0 keycloak 	at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:132)
sso-keycloakx-0 keycloak 	at org.jgroups.protocols.FailureDetection.up(FailureDetection.java:186)
sso-keycloakx-0 keycloak 	at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:254)
sso-keycloakx-0 keycloak 	at org.jgroups.protocols.MERGE3.up(MERGE3.java:281)
sso-keycloakx-0 keycloak 	at org.jgroups.protocols.Discovery.up(Discovery.java:300)
sso-keycloakx-0 keycloak 	at org.jgroups.protocols.TP.passMessageUp(TP.java:1400)
sso-keycloakx-0 keycloak 	at org.jgroups.util.SubmitToThreadPool$SingleMessageHandler.run(SubmitToThreadPool.java:98)
sso-keycloakx-0 keycloak 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
sso-keycloakx-0 keycloak 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
sso-keycloakx-0 keycloak 	at java.base/java.lang.Thread.run(Thread.java:829)

Log on the new (18.0.2) cluster member:

sso-keycloakx-1 keycloak 2022-07-03 17:27:03,948 WARN  [org.infinispan.statetransfer.InboundTransferTask] (jgroups-6,sso-keycloakx-1-38522) ISPN000210: Failed to request state of cache authenticationSessions from node sso-keycloakx-0-46161, segments {0-255}: org.infinispan.commons.CacheException: ExceptionResponse(java.io.EOFException)
sso-keycloakx-1 keycloak 	at org.infinispan.statetransfer.InboundTransferTask.lambda$startTransfer$3(InboundTransferTask.java:167)
sso-keycloakx-1 keycloak 	at java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:930)
sso-keycloakx-1 keycloak 	at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:907)
sso-keycloakx-1 keycloak 	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
sso-keycloakx-1 keycloak 	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073)
sso-keycloakx-1 keycloak 	at org.infinispan.remoting.transport.AbstractRequest.complete(AbstractRequest.java:67)
sso-keycloakx-1 keycloak 	at org.infinispan.remoting.transport.impl.SingleTargetRequest.onResponse(SingleTargetRequest.java:46)
sso-keycloakx-1 keycloak 	at org.infinispan.remoting.transport.impl.RequestRepository.addResponse(RequestRepository.java:51)
sso-keycloakx-1 keycloak 	at org.infinispan.remoting.transport.jgroups.JGroupsTransport.processResponse(JGroupsTransport.java:1495)
sso-keycloakx-1 keycloak 	at org.infinispan.remoting.transport.jgroups.JGroupsTransport.processMessage(JGroupsTransport.java:1397)
sso-keycloakx-1 keycloak 	at org.infinispan.remoting.transport.jgroups.JGroupsTransport.access$300(JGroupsTransport.java:145)
sso-keycloakx-1 keycloak 	at org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.up(JGroupsTransport.java:1582)
sso-keycloakx-1 keycloak 	at org.jgroups.JChannel.up(JChannel.java:780)
sso-keycloakx-1 keycloak 	at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:913)
sso-keycloakx-1 keycloak 	at org.jgroups.protocols.FRAG3.up(FRAG3.java:165)
sso-keycloakx-1 keycloak 	at org.jgroups.protocols.FlowControl.up(FlowControl.java:347)
sso-keycloakx-1 keycloak 	at org.jgroups.protocols.FlowControl.up(FlowControl.java:347)
sso-keycloakx-1 keycloak 	at org.jgroups.protocols.pbcast.GMS.up(GMS.java:876)
sso-keycloakx-1 keycloak 	at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:254)
sso-keycloakx-1 keycloak 	at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1048)
sso-keycloakx-1 keycloak 	at org.jgroups.protocols.UNICAST3.addMessage(UNICAST3.java:771)
sso-keycloakx-1 keycloak 	at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:752)
sso-keycloakx-1 keycloak 	at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:405)
sso-keycloakx-1 keycloak 	at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:592)
sso-keycloakx-1 keycloak 	at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:132)
sso-keycloakx-1 keycloak 	at org.jgroups.protocols.FailureDetection.up(FailureDetection.java:186)
sso-keycloakx-1 keycloak 	at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:254)
sso-keycloakx-1 keycloak 	at org.jgroups.protocols.MERGE3.up(MERGE3.java:281)
sso-keycloakx-1 keycloak 	at org.jgroups.protocols.Discovery.up(Discovery.java:300)
sso-keycloakx-1 keycloak 	at org.jgroups.protocols.TP.passMessageUp(TP.java:1400)
sso-keycloakx-1 keycloak 	at org.jgroups.util.SubmitToThreadPool$SingleMessageHandler.run(SubmitToThreadPool.java:98)
sso-keycloakx-1 keycloak 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
sso-keycloakx-1 keycloak 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
sso-keycloakx-1 keycloak 	at java.base/java.lang.Thread.run(Thread.java:829)

[stianst]

With Keycloak 20 the WildFly based distribution is no longer supported. For the newer Quarkus distribution of Keycloak, check out the new documentation, or the updated container sources.