carrefour-group / elastic-phenix-operator Goto Github PK
View Code? Open in Web Editor NEWelastic-phenix-operator is a kubernetes operator to manage elasticsearch resources: Indices, Templates, ...
License: Apache License 2.0
elastic-phenix-operator is a kubernetes operator to manage elasticsearch resources: Indices, Templates, ...
License: Apache License 2.0
In ValidatingWebhook
at creation time check model
field json validation (for elasticindex
or elastictemplate
):
_, err := (&utils.EsModel{Model: *r.Spec.Model}).IsValid(utils.Index)
But at update time it doesn't check. So if we create a valid json model
, and we update it later with a corrupted json model
=> the error will not be catched by ValidatingWebhook#ValidateUpdate
for ex:
response, err := esapi.IndicesPutTemplateRequest{Name: templateName, Body: strings.NewReader(model), IncludeTypeName: shouldIncludeTypeName}.Do(context.Background(), es.Client)
we shoud use a context.WithTimeout instead of context.Background to avoir longrunning http requests
We cannot customize order
value for elasticsearch templates, so we can add a new property in ElasticTemplate
kind.
Elasticsearch template order definition: Order in which Elasticsearch applies this template if index matches multiple templates.
This is an example of an ElasticTemplate
with order
property:
apiVersion: elastic.carrefour.com/v1alpha1
kind: ElasticTemplate
metadata:
name: invoice-template
spec:
templateName: invoice
elasticURI:
secretKeyRef:
name: elasticsearch-cluster-secret
key: uri
numberOfShards: 5
numberOfReplicas: 3
order: 1
model: |-
{
"index_patterns": ["invoice*"],
"settings": {
},
"mappings": {
"properties": {
"key": {
"type": "keyword",
"index": true
}
}
}
}
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
/workspace/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:55 +0x105
panic(0x1cc21a0, 0x329aed0)
/usr/local/go/src/runtime/panic.go:679 +0x1b2
github.com/Carrefour-Group/elastic-phenix-operator/pkg/utils.(*Elasticsearch7).updateIndexSettings(0xc0018c27c0, 0x21fd5a0, 0xc0018ea9c0, 0xc000499f20, 0xf, 0xc
000812240, 0x228, 0x0, 0x18, 0x1f34d8b)
/workspace/pkg/utils/elasticsearch7.go:214 +0x168
github.com/Carrefour-Group/elastic-phenix-operator/pkg/utils.(*Elasticsearch7).CreateOrUpdateIndex(0xc0018c27c0, 0x21fd5a0, 0xc0018ea9c0, 0xc000499f20, 0xf, 0xc
000812240, 0x228, 0x0, 0x0, 0x0)
/workspace/pkg/utils/elasticsearch7.go:164 +0x234
github.com/Carrefour-Group/elastic-phenix-operator/pkg/controllers.(*ElasticIndexReconciler).Reconcile(0xc0000bb140, 0xc000499ee0, 0xe, 0xc000a96200, 0x1f, 0xc0
0038bcd8, 0xc0007ee240, 0xc00098e768, 0x21d3a40)
/workspace/pkg/controllers/elasticindex_controller.go:86 +0xb06
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc0001be000, 0x1d12e00, 0xc00000de60, 0x43ec00)
When we delete a field from model
properties level 1 of an elasticindex
object, we get an Error
status in the elasticindex
object Status
.
But when we delete a field from a nested object properties, the elasticindex
object remains in Created
status, and we expected that it should be in Error
status.
Allow delete indices
and templates
in elasticsearch
cluster using the annotation
: carrefour.com/delete-in-cluster=true
, instead of using the flag enable-delete
in manager level.
e.g. add annotation:
kubectl annotate elasticindex/my-index carrefour.com/delete-in-cluster=true
kubectl annotate elastictemplate/my-template carrefour.com/delete-in-cluster=true
Default value of Webhooks
side-effects field is: sideEffects: Unknown
, so we should define explicitly sideEffects: None
to allow kubectl dry-run
Actually, MutatingWebhookConfiguration
and ValidatingWebhookConfiguration
of elastic-phenix-operator
does not have side-effects
, and when we see kubectl documentation:
"dry-run requests are only processed if all admission controllers explicitly announce that they don't have any dry-run side-effects."
link: https://kubernetes.io/blog/2019/01/14/apiserver-dry-run-and-kubectl-diff/#apiserver-dry-run
When using ElasticSearch 8 index creation fails with:
"http-response": "[400 Bad Request] {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"request [/test-index] contains unrecognized parameter: [include_type_name]"}],"type":"illegal_argument_exception","reason":"request [/test-index] contains unrecognized parameter: [include_type_name]"},"status":400}"}
When creating an elastictemplate
with type
in mappings, the template was not created and I get a status Error
in kubernetes (for elasticindex
it does work with or without type):
The elastictemplate
yaml:
apiVersion: elastic.carrefour.com/v1alpha1
kind: ElasticTemplate
metadata:
name: invoice-template
namespace: elasticsearch
spec:
templateName: invoice
elasticURI:
secretKeyRef:
name: elasticsearch-cluster-secret
key: uri
numberOfShards: 5
numberOfReplicas: 3
model: |-
{
"index_patterns": ["invoice*"],
"settings": {
},
"mappings": {
"_doc": {
"_source": {
"enabled": true
},
"properties": {
"key": {
"type": "keyword",
"index": true
}
}
}
}
}
The error I get in kubernetes elastictemplate STATUS
:
> kubectl get elastictemplate --all-namespaces
NAMESPACE NAME TEMPLATE_NAME SHARDS REPLICAS STATUS AGE
elasticsearch invoices-template invoice 5 3 Error 42s
> kubectl describe elastictemplate/invoices-template -n elasticsearch
Name: invoices-template
Namespace: elasticsearch
Labels: env=dev
Annotations: API Version: elastic.carrefour.com/v1alpha1
Kind: ElasticTemplate
Metadata:
...
Spec:
...
Status:
Http Code Status: 400
Message: [400 Bad Request] {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"The mapping definition cannot be nested under a type [_doc] unless include_type_name is set to true."}],"type":"illegal_argument_exception","reason":"The mapping definition cannot be nested under a type [_doc] unless include_type_name is set to true."},"status":400}
Status: Error
Events: <none>
When creating an ElasticTemplate
with multiple errors e.g. the template below with two errors:
1/ use type keywords
instead of keyword
2/ use type texts
instead of text
, so when we create this ElasticTemplate
:
apiVersion: elastic.carrefour.com/v1alpha1
kind: ElasticTemplate
metadata:
name: invoice-template
namespace: elastic-system
spec:
templateName: invoice
elasticURI:
secretKeyRef:
name: elasticsearch-cluster-secret
key: uri
numberOfShards: 5
numberOfReplicas: 3
model: |-
{
"index_patterns": ["invoice*"],
"settings": {
},
"mappings": {
"_doc": {
"_source": {
"enabled": true
},
"properties": {
"key": {
"type": "keywords",
"index": true
},
"content": {
"type": "texts",
"index": true
}
}
}
}
}
The ElasticTemplate
is created with Error
status as expected:
> kubectl describe elastictemplate/invoice-template -n elastic-system
Name: invoice-template
Namespace: elastic-system
Annotations: API Version: elastic.carrefour.com/v1alpha1
Kind: ElasticTemplate
Metadata:
....
Spec:
Elastic URI: ...
Model: ...
Number Of Replicas: 3
Number Of Shards: 5
Template Name: invoice
Status:
Status: Error
Http Code Status: 400
Message: [400 Bad Request] {"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"No handler for type [texts] declared on field [content]"}],"type":"mapper_parsing_exception","reason":"Failed to parse mapping [_doc]: No handler for type [texts] declared on field [content]","caused_by":{"type":"mapper_parsing_exception","reason":"No handler for type [texts] declared on field [content]"}},"status":400}
When we fix this error (fix type texts
by using elasticsearch type text
), we got the same error message instead of getting a new error message with the new error:
apiVersion: elastic.carrefour.com/v1alpha1
kind: ElasticTemplate
metadata:
name: invoice-template
namespace: elastic-system
spec:
templateName: invoice
elasticURI:
secretKeyRef:
name: elasticsearch-cluster-secret
key: uri
numberOfShards: 5
numberOfReplicas: 3
model: |-
{
"index_patterns": ["invoice*"],
"settings": {
},
"mappings": {
"_doc": {
"_source": {
"enabled": true
},
"properties": {
"key": {
"type": "keywords",
"index": true
},
"content": {
"type": "text",
"index": true
}
}
}
}
}
I can reproduce the same error when creating an ElasticIndex
.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.