Handling for system indices

[zalando-incubator]

Handling for system indices

[otrosien]

@s-vkropotko alternative to the current implementation: Should the ES-Operator move the system indices away (exclude allocation of the system index from the current EDS). They not only shouldn't be scaled up or down but they could also be harming an even balancing out of user shards. WDYT?

[s-vkropotko]

Hi!
According to this elastic/elasticsearch#50251
Smarter replication for system indices (continue to use auto expand 0-1 or maybe a new scheme?)
System indices suppose to use auto_expand_replicas option for now but this can be changed in the future.
Also, users can create indices with this option(auto_expand_replicas) then ES-Op cannot manage a number of replicas for these indices (as the direct update of number_of_replicas acceptable but not applicable).

>> curl http://localhost:9200/_cat/indices
green open test5 Ietp5gGtQcK0WBWsbnnh3g 1 1 0 0 416b 208b
>> curl -X PUT http://localhost:9200/test5/_settings --data '{"settings": {"number_of_replicas": 2}}' -H 'Content-type: application/json'
{"acknowledged":true}
>> curl "http://localhost:9200/test5/_settings?pretty=true"
{
  "test5" : {
    "settings" : {
      "index" : {
        "number_of_shards" : "1",
        "auto_expand_replicas" : "0-1",
        "provided_name" : "test",
...
        "number_of_replicas" : "1",
...

So maybe we can try to update the function GetIndices to return only manageable indices and an additional returns value of unmanageable total shards(primary+replicas), e.g. system indices and indices with auto_expand option ? Then we can use it for proper calculation of shards to nodes ratio.