Shard replica added before statefull set ready

[zalando-incubator]

Shard replica added before statefull set ready

[otrosien]

Hi @AyWa, I see your point. Unfortunately the timing of adding index replicas is hard to get right. If you do it too late ES would have started to relocate indices for no reason, and if you do it too early (as we're doing it now) it will be colocated on a n already crowded node, and then moved over. None of it is ideal.

If you're in a lucky position to have single-shard indices, you might want to experiment with "index.auto_expand_replicas", this delegates the replica setting to ES.

Update: The patch you sent looks reasonable. We'll add it to the next release.

[mikkeloscar]

From the issue it's not completely clear to me what the impact is of the shard replica being added before the ES node is ready. Can you elaborate on the impact? I.e. does ES go into a degraded state because of this?

[AyWa]

From the issue it's not completely clear to me what the impact is of the shard replica being added before the ES node is ready. Can you elaborate on the impact? I.e. does ES go into a degraded state because of this?

Sure, so the impact of the replica being added before the ES node is mainly that it duplicate some shards operations (and so use more cpu / increase latency when cluster is really small), and also it is likely to go over the max shards per node that we set in the configuration

Lets make a simple example:

• 2 existing nodes with 1 shards each (no replica) (max shards 1)
• operator decide to auto scale because CPU is more than 50% (add 2 instance)
• Pod can take few minute to be ready
• the 2 existing nodes will send each other a shards
• New Node are ready. Need to wait for ES to re balance again the shards
• then shards are send to new node

If with the same configuration we wait for the node to be ready before creating the replica, we will not have a time where max shards is bigger than expected, and ES will likely move less shards.

Let me know if you need more information