Comments (13)
The go implementation can survive that node 1 exits. It looks like the issue is that we're not passing an observer here: https://github.com/AsynkronIT/protoactor-kotlin/blob/master/proto-remote/src/main/kotlin/actor/proto/remote/EndpointWriter.kt#L81
from protoactor-kotlin.
Is that related to never getting an EndpointTerminatedEvent on the Event Stream when a remote dies - and consequently not receiving any Terminated messages for watched actors on that remote? With the commented lines just below the above link - it doesn't appear EndpointTerminatedEvent is ever published anywhere in the code, and I think generating Terminated messages for watchers of related actors relies on this event?
from protoactor-kotlin.
So this seems to work:
streamWriter = client.receive(object : StreamObserver<RemoteProtos.Unit> {
override fun onNext(value: RemoteProtos.Unit?) {
//never called
}
override fun onCompleted() {
//never called
}
override fun onError(t: Throwable?) {
when(t) {
is StatusRuntimeException -> {
if ("UNAVAILABLE".equals(t.status.code.name)) {
val terminated: EndpointTerminatedEvent = EndpointTerminatedEvent(address)
EventStream.publish(terminated)
println("Lost connection to address $address")
} else {
println("Stream Status Runtime Exception "+t.toString())
}
}
else -> {
println("Other exception "+t.toString())
}
}
}
})
The EndpointTerminatedEvent is now received by Event Stream subscribers. And watchers of actors on a terminated remote are sent a Terminated message.
Haven't yet tested:
- What happens if the remote endpoint quickly recovers. I don't think there's currently any retry
- What state the termination leaves the remote. So what happens if after the termination a message is sent to an actor on the terminated remote. Is the expected behaviour that this would cause a DeadLetter event? Or is another connection attempt expected?
from protoactor-kotlin.
@james-cobb nice drill down of the problem! @rogeralsing would you mind having a look at the questions?
from protoactor-kotlin.
What state the termination leaves the remote. So what happens if after the termination a message is sent to an actor on the terminated remote. Is the expected behaviour that this would cause a DeadLetter event? Or is another connection attempt expected?
In the Go and C# impl, the EndpintWatcher actor will detect the termination and trigger a Terminated event to be sent to all subscribers of a remote actors lifecycles.
That is, if a local actor has used context.Watch(remotePID)
that local actor will get a Terminated message with the remote PID if the endpoint breaks.
from protoactor-kotlin.
https://github.com/AsynkronIT/protoactor-dotnet/blob/dev/src/Proto.Remote/EndpointWatcher.cs#L59
I'm not sure in what state the Kotlin impl is in this regards, me and @potterdai have done some work on both Go and C# in this specific regard.
Also when it comes to re-trying connections
from protoactor-kotlin.
Thanks for the responses. In digging into this I've found a couple of other issues that were causing problems for the EndpointWriter - I'll report as separate issues on github, then circle back on this one.
from protoactor-kotlin.
from protoactor-kotlin.
@orjan thanks! I think I'll be comfortable to submit some pull requests soon. But need some guidance on the required behaviour in some of the cases I've been unpicking - see #29 #30 #31
from protoactor-kotlin.
@orjan I'm readying a pull request. In addition to the StreamObserver to catch exceptions from gRPC, I made some of the gRPC options like keepalives configurable. When a node vanishes, it was taking 15 minutes for gRPC to realise, which means 15 minutes of messages sent to remote actors that are silently lost. Using keepAliveTime and keepAliveTimeout we can reduce this to only a few seconds, but I think this needs to be configurable for each application.
Can I check if you're open to the way I've done this, using the RemoteConfig class before I make a PR?
from protoactor-kotlin.
from protoactor-kotlin.
My thinking was not to apply any defaults within the code - if a parameter is not explicitly set in an application using Remote.Config, then it's not set at all when instantiating the gRPC server / client. Which means their defaults are used, and if the gRPC library changes in future releases, we don't need to update default values that have become embedded into proto.actor. That was the reason for everything being nullable. Except usePlainText, which defaults to true simply to make sure that the change is backward compatible. If no settings are specified, the connection will fall back to exactly the current set up.
I'll go ahead with a PR
from protoactor-kotlin.
from protoactor-kotlin.
Related Issues (20)
- DefaultMailbox.processMessages function is bad written on many levels HOT 3
- Actor system can deadlock HOT 1
- Publish to maven central HOT 1
- jcenter integration for proto-mailbox HOT 2
- Actor can receive message before mailbox dispatcher is initialized HOT 5
- UninitializedPropertyAccessException can occur in EndpointWriter restarting() method causing infinite loop HOT 3
- Actor can receive a user message before Started message is successfully processed HOT 2
- Use SLF4J for logging HOT 4
- Mailbox schedule() called in endless loop without any messages HOT 1
- DeferredProcess causes a memory leak HOT 7
- Wrong action from supervisor strategy when running multiple tests
- Is this project dead? HOT 12
- Failing Unit Test HOT 9
- link to documentation 404s
- Release newer version HOT 5
- Remove old branches to avoid confusion HOT 1
- Are our libraries correctly pushed to jcenter? HOT 4
- Java Duration @RequiresApi(Build.VERSION_CODES.O)
- Any chance of this project being revived? HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from protoactor-kotlin.