Comments (3)
This is an Akka.Cluster
behaviour and the short answer is "it depends".
- How big is your cluster
- What is your cluster size VS. seed nodes ratio
- Are you willing to take the risk that a big part of the cluster will be downed if that part was split brained from the smaller chunk of the cluster that has all the seed nodes.
When you set KeepMajority.Role
, what will happen when a split brain occured is that only the cluster members that has that role is being considered when SBR tries to resolve the split. This would mean that you will need at least 5 seed nodes for this to work properly in production.
But lets take some examples:
Cluster settings:
- 5 seed nodes with role "seed"
- 100 nodes with no roles
KeepMajority.Role
is set to "seed"
Scenario 1, the happy path:
The cluster split into these parts:
- Part 1: 3 "seed" nodes and 80 non-role nodes
- Part 2: 2 "seed" nodes and 20 non-role nodes
SBR Resolution: SBR will down part 2
Scenario 2, the not-so-happy path:
The cluster split into these parts:
- Part 1: 3 "seed" nodes and 10 non-role nodes
- Part 2: 2 "seed" nodes and 90 non-role nodes
SBR Resolution: SBR will down part 2, even when it has the "majority" of general node count. This is because SBR only considers the number of the nodes that has majority inside the declared role.
Scenario 3, the ugly path:
- All of the "seed" roles are down, leaving the 100 non-role to be stranded.
- 1 "seed" role node are restarted and self-join itself to form a cluster
SBR Resolution: Nothing. You will end up with a permanent split brain and they could not corellate to each other because the non-role cluster does not know about the newly created cluster.
from akka.net.
The only way to fix this problem is to remove the arbiter inside the cluster, be it a Lighthouse instance or a fixed count of seed nodes. To do this, you will need Akka.Management.Cluster.Bootstrap
in combination with Akka.Discovery
which uses an arbiter outside of the cluster and is available for Kubernetes, Azure, and AWS.
Note that Akka.Discovery.Config
is not the answer. It is still using the cluster itself as the arbiter, which defeats the purpose.
from akka.net.
OK, thanks for your input! Guess our only strategy atm is higher tolerance for heartbeats, to avoid unnecessary disconnects.
from akka.net.
Related Issues (20)
- getakka.net documentation is broken HOT 1
- "System.Runtime.Serialization.SerializationException: Failed to deserialize object of type [Akka.Actor.Status+Failure] from the stream. Cause: Index was outside the bounds of the array." After Akka Upgrade from 1.3.14 to 1.4.48 HOT 31
- [Documention] Problem upgrading to DocFx 2.67.0
- Akka.NET v1.5 upgrade guide: need to add section on logging with more than 6 parameters
- Adjusted frame length exceeds 128000: 542393675 HOT 7
- Akka.Tools.ClusterClient: `ClusterClient.Send` and `ClusterClient.SendToAll` aren't serialized using `ClusterClientSerializer` HOT 3
- [PERF] Backoff Supervisor uses Expression based Props
- [PERF] ActorGraphInterpreter and other Akka streams internals use Expression based props
- Apache Zookeeper support for Akka.Discovery
- Modify Hocon File
- Akka.DistributedData.LightningDb: move durable folder creation outside of actor constructor in order to preserve stack trace / provide clearer error messages
- Delay in message processing HOT 1
- Add `Tags` property to `Akka.Persistence.Query` `EventEnvelope`
- Logs: add `SpanId` and `ActivityId` to each `LogEvent` producer by `ILoggingAdapter` HOT 1
- Akka.Remote.EndpointDisassociatedException: Disassociated HOT 6
- WatchWith PoisonPill.Instance, PoisonPill is not stopping actor HOT 6
- Akka.Streams: `SupervisionStrategy` not being enforced HOT 2
- Akka.Streams: `SelectAsync` stage - none of the exceptions thrown inside the `await` are logged by default
- Source from IAsyncEnumerable is not disposed on shutdown, while interrupted because of back-pressure HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from akka.net.