Remote terminates when connection is lost

[asynkron]

Remote terminates when connection is lost

[orjan]

The go implementation can survive that node 1 exits. It looks like the issue is that we're not passing an observer here: https://github.com/AsynkronIT/protoactor-kotlin/blob/master/proto-remote/src/main/kotlin/actor/proto/remote/EndpointWriter.kt#L81

[james-cobb]

Is that related to never getting an EndpointTerminatedEvent on the Event Stream when a remote dies - and consequently not receiving any Terminated messages for watched actors on that remote? With the commented lines just below the above link - it doesn't appear EndpointTerminatedEvent is ever published anywhere in the code, and I think generating Terminated messages for watchers of related actors relies on this event?

[james-cobb]

So this seems to work:

 streamWriter = client.receive(object : StreamObserver<RemoteProtos.Unit> {
            override fun onNext(value: RemoteProtos.Unit?) {
                //never called
            }
            override fun onCompleted() {
               //never called
            }
            override fun onError(t: Throwable?) {
                when(t) {
                    is StatusRuntimeException -> {
                        if ("UNAVAILABLE".equals(t.status.code.name)) {
                            val terminated: EndpointTerminatedEvent = EndpointTerminatedEvent(address)
                            EventStream.publish(terminated)
                            println("Lost connection to address $address")
                        } else {
                            println("Stream Status Runtime Exception "+t.toString())
                        }
                    }
                    else -> {
                        println("Other exception "+t.toString())
                    }
                }
            }
        })

The EndpointTerminatedEvent is now received by Event Stream subscribers. And watchers of actors on a terminated remote are sent a Terminated message.

Haven't yet tested:

• What happens if the remote endpoint quickly recovers. I don't think there's currently any retry
• What state the termination leaves the remote. So what happens if after the termination a message is sent to an actor on the terminated remote. Is the expected behaviour that this would cause a DeadLetter event? Or is another connection attempt expected?

[orjan]

@james-cobb nice drill down of the problem! @rogeralsing would you mind having a look at the questions?

[rogeralsing]

What state the termination leaves the remote. So what happens if after the termination a message is sent to an actor on the terminated remote. Is the expected behaviour that this would cause a DeadLetter event? Or is another connection attempt expected?

In the Go and C# impl, the EndpintWatcher actor will detect the termination and trigger a Terminated event to be sent to all subscribers of a remote actors lifecycles.

That is, if a local actor has used context.Watch(remotePID) that local actor will get a Terminated message with the remote PID if the endpoint breaks.

[rogeralsing]

https://github.com/AsynkronIT/protoactor-dotnet/blob/dev/src/Proto.Remote/EndpointWatcher.cs#L59

I'm not sure in what state the Kotlin impl is in this regards, me and @potterdai have done some work on both Go and C# in this specific regard.
Also when it comes to re-trying connections

[james-cobb]

Thanks for the responses. In digging into this I've found a couple of other issues that were causing problems for the EndpointWriter - I'll report as separate issues on github, then circle back on this one.

[orjan]

@james-cobb great initiative, feel free to submit pull requests if you want to! // Örjan

…

On Mon, Jan 15, 2018 at 8:38 PM, james-cobb ***@***.***> wrote: Thanks for the responses. In digging into this I've found a couple of other issues that were causing problems for the EndpointWriter - I'll report as separate issues on github, then circle back on this one. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#27 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAHkgKFYw1CB9mxqN2M5Y0WechbhquWnks5tK6lGgaJpZM4RPtnM> .

-- Örjan Sjöholm 0727 - 130037

[james-cobb]

@orjan thanks! I think I'll be comfortable to submit some pull requests soon. But need some guidance on the required behaviour in some of the cases I've been unpicking - see #29 #30 #31

[james-cobb]

@orjan I'm readying a pull request. In addition to the StreamObserver to catch exceptions from gRPC, I made some of the gRPC options like keepalives configurable. When a node vanishes, it was taking 15 minutes for gRPC to realise, which means 15 minutes of messages sent to remote actors that are silently lost. Using keepAliveTime and keepAliveTimeout we can reduce this to only a few seconds, but I think this needs to be configurable for each application.

Can I check if you're open to the way I've done this, using the RemoteConfig class before I make a PR?

https://github.com/crowdconnected/protoactor-kotlin/blob/4a9565735213b43d26998dc05677d3d2b4b8ca82/proto-remote/src/main/kotlin/actor/proto/remote/RemoteConfig.kt

[orjan]

I would say go ahead and submit a PR! 2 small concerns: - Does all fields really need to be nullable - Can we apply sensible default values

…

On Thu, Jan 18, 2018 at 12:47 PM, james-cobb ***@***.***> wrote: @orjan <https://github.com/orjan> I'm readying a pull request. In addition to the StreamObserver to catch exceptions from gRPC, I made some of the gRPC options like keepalives configurable. When a node vanishes, it was taking 15 minutes for gRPC to realise, which means 15 minutes of messages sent to remote actors that are silently lost. Using keepAliveTime and keepAliveTimeout we can reduce this to only a few seconds, but I think this needs to be configurable for each application. Can I check if you're open to the way I've done this, using the RemoteConfig class before I make a PR? https://github.com/crowdconnected/protoactor-kotlin/blob/ 4a95657/proto-remote/src/ main/kotlin/actor/proto/remote/RemoteConfig.kt — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#27 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAHkgKLnMM6GWIJmhAuFkXQej92d0YTwks5tLy9mgaJpZM4RPtnM> .

-- Örjan Sjöholm 0727 - 130037

[james-cobb]

My thinking was not to apply any defaults within the code - if a parameter is not explicitly set in an application using Remote.Config, then it's not set at all when instantiating the gRPC server / client. Which means their defaults are used, and if the gRPC library changes in future releases, we don't need to update default values that have become embedded into proto.actor. That was the reason for everything being nullable. Except usePlainText, which defaults to true simply to make sure that the change is backward compatible. If no settings are specified, the connection will fall back to exactly the current set up.

I'll go ahead with a PR

[orjan]

Sorry my bad! You're right, it's better to use the default from gRPC!

…

On Fri, Jan 19, 2018 at 9:46 AM, james-cobb ***@***.***> wrote: My thinking was not to apply any defaults within the code - if a parameter is not explicitly set in an application using Remote.Config, then it's not set at all when instantiating the gRPC server / client. Which means their defaults are used, and if the gRPC library changes in future releases, we don't need to update default values that have become embedded into proto.actor. That was the reason for everything being nullable. Except usePlainText, which defaults to true simply to make sure that the change is backward compatible. If no settings are specified, the connection will fall back to exactly the current set up. I'll go ahead with a PR — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#27 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAHkgAqhBZZV4Zf5y6BO5Ff6HDmKKPogks5tMFZrgaJpZM4RPtnM> .

-- Örjan Sjöholm 0727 - 130037