Comments (2)
Are you running Coherence directly on the AWS VMs or are you running in some form of Kubernetes cluster? In Kubernetes we have the Coherence Operator which will safely manage scaling of clusters and can hook into the Kubernetes horizontal Pod autoscaler. If you are running Coherence directly on the AWS VMs then you need to manage scaling yourself.
When scaling down, a lot of care needs to be taken to ensure there is no data loss. When trying to hook into autoscalers, this can sometimes be difficult. For example a typical auto-scaler will just scale down by a specific number of processes, e.g. you have a cluster of 50 members, the autoscaler decides you only need 40 and kills 10. This is bad news for Coherence as that will almost certainly cause data loss. If you know your cache services are site safe, rack safe or machine safe, then you can kill all members in a given site, rack, or machine without data loss, but most autoscalers do not do this - and actually you probably don't want that either. When using Coherence in k8s, the autoscaler scales down by a given number, but the Coherence Operator actually controls the scale down under the covers and safely scales down, one member at at time, with safety/health checks between each member removal.
There is nothing in Coherence where a Coherence member can signal that it is shutting down. Even if it could, there would still need to be a number of checks in place to determine whether it is "safe" for that member to leave the cluster. Those checks depend on a number of things, some of which can be application specific.
In the next release of Coherence (both CE and commercial) due in June there is a health check API that allows applications to determine whether it is safe to remove a member. There is also a health http endpoint that can be used to check health. I have no idea how the AWS auto scaling group works or what checks it allows, but maybe it can hook into the new health check to control when it scales down.
from coherence.
No we are not seeing any value in our use-case for containerizing our Coherence cache nodes (in particular not with Kubernetes) so sadly we have to roll our own solution to this as you sugests. An API for health check sounds like a good idea - looking forward to when it is released! When only allowing auto scaling to remove ONE instance at a time the API sounds like it could work well for use in hooks that delay the node from leaving until the health status allow it...
from coherence.
Related Issues (20)
- PagedTopic MBean attributes documentation
- Bad performance pruning large caches HOT 4
- Is there a function to obtain hotkeys implemented HOT 3
- Restrictive license text found in one of the versions of Coherence HOT 5
- Multicast error message on MacOS HOT 6
- Problem with configurating invocation service HOT 3
- How to do grid wide processing per partition? HOT 18
- How to configure the same partitioned cache to be a back cache both for use "in cluster" and with gRPC? HOT 14
- Detail question regarding StreamingAggregator methods and characteristics HOT 3
- Thoughts on running Coherence cluster over multiple availability zones in the cloud... HOT 4
- Possible to insert data directly to front cache without replication to back cache? HOT 2
- Dynamic backups
- Backups in Flash
- Provide a Graddle alternative for Coherence POF Maven plugin HOT 1
- CaffeineCache doesn't track entry and cache size correctly when unit factor is used HOT 1
- Relationship between future releases of Coherence Enterprise/Grid and Coherence Community HOT 3
- COH-23644? HOT 9
- Support to change the percentage used for calculating default low-units HOT 1
- Is "-Dtangosol.coherence.override" not used in Coherence CE? Still talking about the old v14 sorry for that.... :-( HOT 7
- initiating connection migration with tmb? HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from coherence.