shmel1k/qumomf

Mom's friend vshard quorum

qumomf's Issues

Anti-flapping protection

Avoid flapping (cascading failures causing continuous outage and elimination of resources) by introducing a block period, where on any given replica set, qumomf will not kick in automated recovery on an interval smaller than said period.

Cover main logic by unit and integration tests

Just will be really useful in future development process.

Write some documentation

1. README
2. License
3. Maybe something else

Build deb package

Will be easy to deploy and distribute

Move logger level to config

Default level is debug.

Implement auto discovery of cluster

Using vshard.storage.info or vshard.router.info we might auto discover the cluster topology. User would just provide credentials and initial endpoint to connect.

Each cluster should have a unique name to identify them. Names and other meta information we can store to a database backend like SQLite.

API

What should be implemented:

• Building a cluster topology (Cluster structure and basic data on nodes).
• Viewing information on a separate node
• Getting a list of recoveries
• Getting information on individual recovery
• Getting a list of active problems

Filter out replicas with lag more than user specified value

We should provide an options to filter out replicas which are too lagging from master election:

1. Max idle value, in seconds
2. Max LSN behind the master.

It will allow users to control the master election behaviour and fail the promotion if chosen replica is too far from master.

Make a decision to run failover using consensus

The idea is similar to the logic realized in MySQL Orchestrator.

Current implementation: qumomf decides to switch replica role to master if it lost the connection to old master. It might lead to a false failover.

I suggest to analyze replica alerts and if they notify that replication is broken too then qumomf should start a failover process.

Integrate to CI

Automate build, test, lint tasks

More readable metrics

Add instance URI to metrics. UUID is not handy when looking for a problem.

Add readonly option

For testing purpose it would be useful to have a readonly option. If that option is enabled, qumomf will not change cluster topology in any way.

Implement better logging

Replace log + fmt to something more convenient, e.g. zerolog

Save cluster snapshots and recoveries in SQLite

It will allow to save and retrieve all important information in case of failures.

Add health check

To monitor the qumomf status and getting its info like version, built time, etc.

Improve logging

• lower the log level for non-essential logs to the debug level
• log only in case of a change in the topology or state of the cluster
• log instance URI

Notify about concrete dead follower

Subj.

{"level":"warn","cluster":"order_control","replica_set":"5065fb5f-5f40-498e-af79-43887ba3d1ec","master_uri":"some_master_uri", ,"message":"Master is reachable but some of its replicas are not replicating. No actions will be applied."}

Can you please add some additional information about replicas that are failing to replicate?

Define manual promotion priority to each replica

Let users to specify custom promotion priorities to each replica.

Negative priority should mean that replica must not participate in the master election.

Timestamp to logs

Subj. No timestamp is currently present in logs. :(