carrefour-group/elastic-phenix-operator

elastic-phenix-operator is a kubernetes operator to manage elasticsearch resources: Indices, Templates, ...

elastic-phenix-operator's Issues

ValidateUpdate in ValidatingWebhook does not check model json validation

In ValidatingWebhook at creation time check model field json validation (for elasticindex or elastictemplate):

_, err := (&utils.EsModel{Model: *r.Spec.Model}).IsValid(utils.Index)

But at update time it doesn't check. So if we create a valid json model, and we update it later with a corrupted json model => the error will not be catched by ValidatingWebhook#ValidateUpdate

Use WithTimeout context when call elasticsearch api methods

for ex:
response, err := esapi.IndicesPutTemplateRequest{Name: templateName, Body: strings.NewReader(model), IncludeTypeName: shouldIncludeTypeName}.Do(context.Background(), es.Client)

we shoud use a context.WithTimeout instead of context.Background to avoir longrunning http requests

Cannot customize order property in ElasticTemplate Kind

We cannot customize order value for elasticsearch templates, so we can add a new property in ElasticTemplate kind.
Elasticsearch template order definition: Order in which Elasticsearch applies this template if index matches multiple templates.
This is an example of an ElasticTemplate with order property:

apiVersion: elastic.carrefour.com/v1alpha1
kind: ElasticTemplate
metadata:
  name: invoice-template
spec:
  templateName: invoice
  elasticURI:
    secretKeyRef:
      name: elasticsearch-cluster-secret
      key: uri
  numberOfShards: 5
  numberOfReplicas: 3
  order: 1
  model: |-
    {
      "index_patterns": ["invoice*"],
      "settings": {
      },
      "mappings": {
        "properties": {
          "key": {
            "type": "keyword",
            "index": true
          }
        }
      }
    }

issue: when oldNumShards is nil

k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)                                                                                                 
        /workspace/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:55 +0x105                                                                             
panic(0x1cc21a0, 0x329aed0)                                                                                                                                     
        /usr/local/go/src/runtime/panic.go:679 +0x1b2                                                                                                           
github.com/Carrefour-Group/elastic-phenix-operator/pkg/utils.(*Elasticsearch7).updateIndexSettings(0xc0018c27c0, 0x21fd5a0, 0xc0018ea9c0, 0xc000499f20, 0xf, 0xc
000812240, 0x228, 0x0, 0x18, 0x1f34d8b)                                                                                                                         
        /workspace/pkg/utils/elasticsearch7.go:214 +0x168                                                                                                       
github.com/Carrefour-Group/elastic-phenix-operator/pkg/utils.(*Elasticsearch7).CreateOrUpdateIndex(0xc0018c27c0, 0x21fd5a0, 0xc0018ea9c0, 0xc000499f20, 0xf, 0xc
000812240, 0x228, 0x0, 0x0, 0x0)                                                                                                                                
        /workspace/pkg/utils/elasticsearch7.go:164 +0x234                                                                                                       
github.com/Carrefour-Group/elastic-phenix-operator/pkg/controllers.(*ElasticIndexReconciler).Reconcile(0xc0000bb140, 0xc000499ee0, 0xe, 0xc000a96200, 0x1f, 0xc0
0038bcd8, 0xc0007ee240, 0xc00098e768, 0x21d3a40)                                                                                                                
        /workspace/pkg/controllers/elasticindex_controller.go:86 +0xb06                                                                                         
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc0001be000, 0x1d12e00, 0xc00000de60, 0x43ec00) 

Delete fields from nested objects allowed for elasticindex

When we delete a field from model properties level 1 of an elasticindex object, we get an Error status in the elasticindex object Status.
But when we delete a field from a nested object properties, the elasticindex object remains in Created status, and we expected that it should be in Error status.

Delete resources in elasticsearch cluster using annotation

Allow delete indices and templates in elasticsearch cluster using the annotation: carrefour.com/delete-in-cluster=true, instead of using the flag enable-delete in manager level.

e.g. add annotation:

kubectl annotate elasticindex/my-index carrefour.com/delete-in-cluster=true
kubectl annotate elastictemplate/my-template carrefour.com/delete-in-cluster=true

OutOfMemory in manager container when using default mem/cpu limit

Manager container is generally in OutOfMemory when using default memory & cpu request/limit values.
Should add more memory for default values:

        name: manager
        resources:
          limits:
            cpu: 100m
            memory: 30Mi
          requests:
            cpu: 100m
            memory: 20Mi

Cannot run kubectl dry-run due to Webhooks side-effects Unknown

Default value of Webhooks side-effects field is: sideEffects: Unknown, so we should define explicitly sideEffects: None to allow kubectl dry-run

Actually, MutatingWebhookConfiguration and ValidatingWebhookConfiguration of elastic-phenix-operator does not have side-effects, and when we see kubectl documentation:
"dry-run requests are only processed if all admission controllers explicitly announce that they don't have any dry-run side-effects."

link: https://kubernetes.io/blog/2019/01/14/apiserver-dry-run-and-kubectl-diff/#apiserver-dry-run

Add support for ElasticSearch 8

When using ElasticSearch 8 index creation fails with:

"http-response": "[400 Bad Request] {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"request [/test-index] contains unrecognized parameter: [include_type_name]"}],"type":"illegal_argument_exception","reason":"request [/test-index] contains unrecognized parameter: [include_type_name]"},"status":400}"}

Error while creating templates with type in mappings

When creating an elastictemplate with type in mappings, the template was not created and I get a status Error in kubernetes (for elasticindex it does work with or without type):

The elastictemplate yaml:

apiVersion: elastic.carrefour.com/v1alpha1
kind: ElasticTemplate
metadata:
  name: invoice-template
  namespace: elasticsearch
spec:
  templateName: invoice
  elasticURI:
    secretKeyRef:
      name: elasticsearch-cluster-secret
      key: uri
  numberOfShards: 5
  numberOfReplicas: 3
  model: |-
    {
      "index_patterns": ["invoice*"],
      "settings": {
      },
      "mappings": {
        "_doc": {
          "_source": {
            "enabled": true
          },
          "properties": {
            "key": {
              "type": "keyword",
              "index": true
            }
          }
        }
      }
    }

The error I get in kubernetes elastictemplate STATUS:

> kubectl get elastictemplate --all-namespaces
NAMESPACE        NAME            TEMPLATE_NAME   SHARDS   REPLICAS   STATUS   AGE
elasticsearch   invoices-template   invoice         5        3       Error    42s

> kubectl describe elastictemplate/invoices-template -n elasticsearch
Name:         invoices-template
Namespace:    elasticsearch
Labels:       env=dev
Annotations:  API Version:  elastic.carrefour.com/v1alpha1
Kind:         ElasticTemplate
Metadata:
  ...
Spec:
  ...
Status:
  Http Code Status:  400
  Message:           [400 Bad Request] {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"The mapping definition cannot be nested under a type [_doc] unless include_type_name is set to true."}],"type":"illegal_argument_exception","reason":"The mapping definition cannot be nested under a type [_doc] unless include_type_name is set to true."},"status":400}
  Status:            Error
Events:              <none>

The error message remains unchanged in an ElasticTemplate or an ElasticIndex

When creating an ElasticTemplate with multiple errors e.g. the template below with two errors:

1/ use type keywords instead of keyword
2/ use type texts instead of text, so when we create this ElasticTemplate:

apiVersion: elastic.carrefour.com/v1alpha1
kind: ElasticTemplate
metadata:
  name: invoice-template
  namespace: elastic-system
spec:
  templateName: invoice
  elasticURI:
    secretKeyRef:
      name: elasticsearch-cluster-secret
      key: uri
  numberOfShards: 5
  numberOfReplicas: 3
  model: |-
    {
      "index_patterns": ["invoice*"],
      "settings": {
      },
      "mappings": {
        "_doc": {
          "_source": {
            "enabled": true
          },
          "properties": {
            "key": {
              "type": "keywords",
              "index": true
            },
            "content": {
              "type": "texts",
              "index": true
            }
          }
        }
      }
    }

The ElasticTemplate is created with Error status as expected:

> kubectl describe elastictemplate/invoice-template -n  elastic-system

Name:         invoice-template
Namespace:    elastic-system
Annotations:  API Version:  elastic.carrefour.com/v1alpha1
Kind:         ElasticTemplate
Metadata:
  ....
Spec:
  Elastic URI: ...
  Model: ...
  Number Of Replicas:  3
  Number Of Shards:    5
  Template Name:       invoice
Status:
  Status:            Error
  Http Code Status:  400
  Message:           [400 Bad Request] {"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"No handler for type [texts] declared on field [content]"}],"type":"mapper_parsing_exception","reason":"Failed to parse mapping [_doc]: No handler for type [texts] declared on field [content]","caused_by":{"type":"mapper_parsing_exception","reason":"No handler for type [texts] declared on field [content]"}},"status":400}

When we fix this error (fix type texts by using elasticsearch type text), we got the same error message instead of getting a new error message with the new error:

apiVersion: elastic.carrefour.com/v1alpha1
kind: ElasticTemplate
metadata:
  name: invoice-template
  namespace: elastic-system
spec:
  templateName: invoice
  elasticURI:
    secretKeyRef:
      name: elasticsearch-cluster-secret
      key: uri
  numberOfShards: 5
  numberOfReplicas: 3
  model: |-
    {
      "index_patterns": ["invoice*"],
      "settings": {
      },
      "mappings": {
        "_doc": {
          "_source": {
            "enabled": true
          },
          "properties": {
            "key": {
              "type": "keywords",
              "index": true
            },
            "content": {
              "type": "text",
              "index": true
            }
          }
        }
      }
    }

I can reproduce the same error when creating an ElasticIndex.