Auto-bootstrapping an all-down cluster

[smellbee]

Auto-bootstrapping an all-down cluster

[sanikolaev]

and client could not know about it

There is a safeguard in place: you must write to cluster:table_name, not just table_name. Therefore, if your cluster is entirely down, you won't be able to write to any table within it. Reading will still function, which is acceptable.

we need to check the seqno, but this is not safe because of the first reason, unexpected data bugs could happen.

Could you elaborate on this? What specific data bugs are you referring to? Isn't the aforementioned protection effective in your case?

so, Auto-bootstrapping an all-down cluster is quite needed, With this feature, we dare to actually use manticore in a production environment

I agree that it would be beneficial if Manticore could handle auto-bootstrapping an entirely down cluster. When designing the replication, our assumption was that the scenario where most nodes go down simultaneously is extremely rare. If this is a frequent issue for you, could you explain what causes it? In my two decades of IT experience, I've only encountered a few instances where an entire datacenter went down.

[smellbee]

There is a safeguard in place: you must write to cluster:table_name, not just table_name.

Thank u for pointing it out, this safeguard is necessary. but it still needs manual intervention.

Manual intervention is the key problem:
firstly, the cluster could not get back on track in the first place, and it often takes long time for feedback to be received before investigations are initiated and solutions submitted.
secondly, the cost of manual intervention is high, involving user feedback, customer service and technical communication, problem troubleshooting, etc.

When designing the replication, our assumption was that the scenario where most nodes go down simultaneously is extremely rare.

Yes, but the key problem is power outages:

Not all companies place their projects in expensive, high-level data centers. Private server rooms are also a common solution for cost savings, or special security needs.
Customers are particularly concerned about the cost of hardware deployment, and they want to use the company's internal network, which can save a significant amount of money.

Therefore, it is very possible for abnormal power outages to occur several times a year, and we have already encountered such situations in actual operations and maintenance. And I really think it's a pain here, if we dont have Auto-bootstrapping an all-down cluster feature