clockworksoul/helm-elasticsearch

An Elasticsearch cluster on top of Kubernetes, made easier, with Helm.

This project is no longer maintained

Dear friends and collegues. Given that the pires/kubernetes-elasticsearch-cluster repository -- the source of truth for this project -- is no longer maintained, and given that I don't have any time or energy to maintain this repository either, I've decided to cease maintaining it. I'll now archive it in hope that it can still serve some historical value.

Thank you all for your help and your support!

helm-elasticsearch

An Elasticsearch cluster on top of Kubernetes, made easier.

A Helm chart that essentially lifts-and-shifts the core manifests in the pires/kubernetes-elasticsearch-cluster project.

Deploying with Helm

With Helm properly installed and configured, standing up a complete cluster is almost trivial:

$ git clone https://github.com/clockworksoul/helm-elasticsearch.git elasticsearch
$ helm install elasticsearch

Contributing

Please do! Taking pull requests.

helm-elasticsearch's Issues

Curator fails when no index matches pattern

This is a BUG REPORT

Version of Helm and Kubernetes:

$ helm version
Client: &version.Version{SemVer:"v2.8.2", GitCommit:"a80231648a1473929271764b920a8e346f6de844", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.8.2", GitCommit:"a80231648a1473929271764b920a8e346f6de844", GitTreeState:"clean"}
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.3", GitCommit:"d2835416544f298c919e2ead3be3d0864b52323b", GitTreeState:"clean", BuildDate:"2018-02-07T12:22:21Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.0", GitCommit:"fc32d2f3698e36b93322a3465f63a14e9f0eaead", GitTreeState:"clean", BuildDate:"2018-03-26T16:44:10Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

What happened:
I deployed this chart with Curator enabled, but I don't have any log indexes yet.

What you expected to happen:
I expect Curator CronJob to finish successfully even if no index matches pattern

Anything else we need to know:

Curator has an ignore_empty_list option which helps to resolve this issue. My working action looks like this:

      1:
        action: delete_indices
        description: "Clean up ES by deleting old indices"
        options:
          timeout_override:
          continue_if_exception: False
          disable_action: False
          ignore_empty_list: True # Added this line
        filters:
        - filtertype: age
          source: name
          direction: older
          timestring: '%Y.%m.%d'
          unit: days
          unit_count: 3
          field:
          stats_result:
          epoch:
          exclude: False

Support stateful mode

Per https://github.com/pires/kubernetes-elasticsearch-cluster/tree/master/stateful

I have a rough approximation of the changes needed: https://github.com/calbach/helm-elasticsearch

However, I don't know if it's in scope for this repo to include the stateful version and I'm not sure of the best way of supporting it:

• Separate chart
• Parameterize a single chart to work for stateful or not

bump to 6.2.2

quay.io/pires/docker-elasticsearch-kubernetes:6.2.2_1 is out.

helm chart name generator

Sorry for being stupied but I really can't find what string of code generates this weird chart names? please point me at it, I beg you)

Add support to create annotations on the deployment and statefulsets

This issue was identified as lalamove/helm-elasticsearch#6. Many thanks to that team for pointing this out, and don't be shy, feel free to submit PR's on your fixes!

Currently, there's no way to push down annotations to the deployments/statefulsets, which would be useful to connect the Elasticsearch pods more permanently to the Prometheus Kubernetes autodiscovery plugin.

Overriding elasticsearch.yml

If I want to override elasticsearch.yml (for example, increasing the thread pool queue size), is there an easy way to do so using values.yaml?

pires/kubernetes-elasticsearch-cluster README seems to suggest that I need to mount my own file and add the ES_PATH_CONF env variable. Could you provide some more detail on how I can accomplish that using this helm chart?

Thanks in advance!

Question: What Kubernetes versions should we support?

Hello all. We should probably decide what versions of Kubernetes we officially support.

I'm personally leaning towards 1.6 through 1.8, unless there's a sufficient number of 1.5 clusters in use to justify support of version 1.5?

Add opt-in ingress for Kibana

Here is my Kibana setup for optional ingress:

values.yaml:

ingress:
  enabled: false
  # Used to create an Ingress record.
  hosts:
    # - kibana.local
  annotations:
    # kubernetes.io/ingress.class: nginx
    # kubernetes.io/tls-acme: "true"
  tls:
    # Secrets must be manually created in the namespace.
    # - secretName: kibana-tls
    #   hosts:
    #     - kibana.local

templates/ingress.yaml:

{{- if .Values.ingress.enabled -}}
{{- $serviceName := include "fullname" . -}}
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: {{ template "fullname" . }}
  labels:
    app: {{ template "name" . }}
    chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}
    release: {{ .Release.Name }}
    heritage: {{ .Release.Service }}
  annotations:
    {{- range $key, $value := .Values.ingress.annotations }}
      {{ $key }}: {{ $value | quote }}
    {{- end }}
spec:
  rules:
    {{- range $host := .Values.ingress.hosts }}
    - host: {{ $host }}
      http:
        paths:
          - backend:
              serviceName: {{ $serviceName }}
              servicePort: 80
    {{- end -}}
  {{- if .Values.ingress.tls }}
  tls:
{{ toYaml .Values.ingress.tls | indent 4 }}
  {{- end -}}
{{- end -}}

With a few changes this could be a great addition to this chart:

• move the ingress section under kibana in values.yaml (and of course change the templates as well)
• suffix names in the template with -kibana
• prefix ingress.yaml with kibana-

Normally I would open a PR, but I'm not sure if I can be responsive these days, so it would take some time to do requested changes, etc, so anyone can pick this if @clockworksoul likes the idea.

If not, I will probably do it later.

max file descriptors error

---

BUG REPORT

Default installation in Kubernetes fails with

ERROR: [1] bootstrap checks failed
[1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]

By the way how does init containers with busy box help us to prepare kernel parameters for elastic containers? They are not connected as I see

Add optional support for external load-balancer service

This feature has been requested more than once. Maybe it's time resurrect the discussion?

Add RBAC

Pointed out over at pires/kubernetes-elasticsearch-cluster#140.

Per that issue:

RBAC is now default and will be much more prominent in kube 1.8. Can you provide the RBAC yaml for this elastic search cluster?

This should include a version conditional however, so that the installation doesn't fail if RBAC is not enabled.

liveness and health probes

It'd be great to add liveness and health probes, like:

        livenessProbe:
          httpGet:
            path: /
            port: 9200
          initialDelaySeconds: 5
          periodSeconds: 10
          timeoutSeconds: 5
        readinessProbe:
          httpGet:
            path: /_cluster/health?local=true&timeout=5s
            port: 9200
          initialDelaySeconds: 5
          periodSeconds: 10
          timeoutSeconds: 5

See https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-health.html for more info.

I guess it can only be done if http is enabled.

Add support for action.auto_create_index property

FEATURE REQUEST

Hello, is it possible to add support for action.auto_create_index property? We use your chart but we need to turn off automatic index creation, as described here: https://www.elastic.co/guide/en/elasticsearch/reference/6.3/docs-index_.html

I can't seem to find a way to do this with this chart.

Thanks a lot.

Accepting PRs?

Just to bring it in line with the helm best practices stuff. I dunno when I'd have time for this, but if I do, this will be the canonical helm chart for me.

turn off external IP

I think this chart is setting the external IP. How do I turn it off?

Proposal: enhanced memory limit settings

Currently memory allocation is defined exclusively using the ES_JAVA_OPTS env property, which on memory constrained systems can lead to crash looping as pods are scheduled onto nodes with less available memory than specified in the JVM options, causing a swift and sometimes vague crash due to insufficient memory.

I propose a single value, set individually for each of the els node types, that sets both the ES_JAVA_OPTS property and a resource request value:

env:
- name: ES_JAVA_OPTS
  value: "-Xms{{ .Values.data.requests.memory }}m -Xmx{{ .Values.data.requests.memory }}m"

and

resources:
    requests:
        memory: {{ .Values.data.requests.memory }}Mi

Thoughts?

Flesh out README.md

The README file is depressingly sparse. I haven't had the time or energy to update it, but since it's the face of the project, and the project is starting to get some eyes, I'd like to consider this a priority.

Really, even if we just copy and modify the pires/kubernetes-elasticsearch-cluster README I think that should be fine.

I'll try to take care of this this weekend if nobody beats me to it.

Add Fluentd

Issue Type
FEATURE REQUEST

Description
Adding an optional Fluentd chart makes it easy to achieve a complete initial logging solution. The current chart only configures half of the solution.

Version of Helm and Kubernetes:

$ helm version
Client: &version.Version{SemVer:"v2.8.0", GitCommit:"14af25f1de6832228539259b821949d20069a222", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.8.0", GitCommit:"14af25f1de6832228539259b821949d20069a222", GitTreeState:"clean"}

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.3", GitCommit:"d2835416544f298c919e2ead3be3d0864b52323b", GitTreeState:"clean", BuildDate:"2018-02-09T21:51:54Z", GoVersion:"go1.9.4", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"9+", GitVersion:"v1.9.2-gke.1", GitCommit:"4ce7af72d8d343ea2f7680348852db641ff573af", GitTreeState:"clean", BuildDate:"2018-01-31T22:30:55Z", GoVersion:"go1.9.2b4", Compiler:"gc", Platform:"linux/amd64"}

What happened:
I installed the chart & none of my containers were ingesting logs to ElasticSearch.

What you expected to happen:
I expected my containers' logs to appear in ElasticSearch.

How to reproduce it:

1. Install the chart.
2. Deploy a container that produces logs to stdout and/or stderr.
3. Open Kibana, use * as the index and elect not to use timestamping.
4. Fail to see any logs from the deployed container.

Anything else we need to know:
My PR, based on https://github.com/fluent/fluentd-kubernetes-daemonset/blob/master/fluentd-daemonset-elasticsearch.yaml, makes the user opt-in in order to remain backward compatible with the current behavior.

Clarify differences from incubator/elasticsearch

This release seems more "official": https://github.com/kubernetes/charts/tree/master/incubator/elasticsearch

I'm wondering if there are intentional design differences, or if it's just two independent implementations. At least one caveat to that incubator version I've found is that it's on Elastic 2.4 (helm/charts#1864).

Does not work for AWS

Please provide set up instruction for AWS. I have changed the storage class to gp2 and enable stateful.
The pods for elasticsearch data, client and master is always crashing

Here is the log -

[2018-06-03T18:38:57,348][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [dunking-joey-elasticsearch-data-0] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalArgumentException: No up-and-running site-local (private) addresses found, got [name:lo (lo), name:eth0 (eth0)]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:125) ~[elasticsearch-6.2.3.jar:6.2.3]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:112) ~[elasticsearch-6.2.3.jar:6.2.3]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.2.3.jar:6.2.3]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.2.3.jar:6.2.3]
at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.2.3.jar:6.2.3]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-6.2.3.jar:6.2.3]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:85) ~[elasticsearch-6.2.3.jar:6.2.3]
Caused by: java.lang.IllegalArgumentException: No up-and-running site-local (private) addresses found, got [name:lo (lo), name:eth0 (eth0)]
at org.elasticsearch.common.network.NetworkUtils.getSiteLocalAddresses(NetworkUtils.java:184) ~[elasticsearch-6.2.3.jar:6.2.3]

add coordinating node support

FEATURE REQUEST:
need coordinating node for large clusters

Please add a LICENSE

Without this I cannot use this repo or send PRs due to company policy. Thanks!

Connection refused on Kibana/Client

When port forwarding to my local machine with the following command:

kubectl port-forward --namespace data elasticsearch-elasticsearch-client-6cc495bc6-bvl2r 9200:9200

I seem to be unable to access the service on that port, e.g. curling the cluster health check endpoint. I get a connection refused error from the port-forward command.

The pods are all running and healthy, and are reporting the occasional cluster message so they appear to be able to communicate fine.

Any ideas on how I can interact with the services?

Changing es-master svc to ClusterIP

BUG_REPORT

As I'm using AWS EKS, LoadBalancer type exposes my elastic masters. I don't need that, so I change es-svc, removing type: LoadBalancer, so that type will automatically change to ClusterIP

But deploing fails with strange:

Error: UPGRADE FAILED: Service "elasticsearch" is invalid: [spec.ports[0].nodePort: Forbidden: may not be used when type is 'ClusterIP', spec.ports[1].nodePort: Forbidden: may not be used when type is 'ClusterIP']

But there is no "nodePort" in any part of config files in the repo, where does it takes them from?

Create a CONTRIBUTING.md page

We'd like to encourage folks to contribute to the cause, and a fully-fleshed-out CONTRIBUTING.md page would be a step in the right direction.

Any takers?

Host helm repo on GitHub Pages

It would be great if there would be helm repo to use for this chart.

You could use TravisCI to package and push it to GitHub Pages.

https://github.com/kubenow/helm-charts is doing this. There you can also find a .travis.yml file which you could copy. (same OSS license)

String comparison in version detection

This is a BUG REPORT

On Kubernetes 1.10.0 I got a problem with version detection. I have .Capabilities.KubeVersion.Minor equal to "10" and this is less than "8" by rules of Golang. We need to convert minor version to integer and then compare.

This line works fine for me

{{- if ge (int (.Capabilities.KubeVersion.Minor)) 8 -}}

Version of Helm and Kubernetes:
Kubernetes 1.10.0
Helm 2.8.2

Master nodes need statefulset support

The latest changes for statefulset support only add it for Elasticsearch data nodes. However, Elasticsearch cluster state is stored across all of nodes, especially the master nodes. In order to do stateful elasticsearch well, we need the same changes to the master node templates.

Please refer to this link for information:
https://discuss.elastic.co/t/where-cluster-metadata-is-stored/44540

Cannot discover other nodes

When deploying the chart:

helm install --name my-release helm-elasticsearch

I see the following errors on all pods:

[2017-07-19T13:40:23,055][WARN ][o.e.d.z.UnicastZenPing   ] [my-release-elasticsearch-master-2563649885-55b98] failed to resolve host [my-release-elasticsearch-discovery]
java.net.UnknownHostException: my-release-elasticsearch-discovery
	at java.net.InetAddress.getAllByName0(InetAddress.java:1280) ~[?:1.8.0_131]
	at java.net.InetAddress.getAllByName(InetAddress.java:1192) ~[?:1.8.0_131]
	at java.net.InetAddress.getAllByName(InetAddress.java:1126) ~[?:1.8.0_131]
	at org.elasticsearch.transport.TcpTransport.parse(TcpTransport.java:921) ~[elasticsearch-5.5.0.jar:5.5.0]
	at org.elasticsearch.transport.TcpTransport.addressesFromString(TcpTransport.java:876) ~[elasticsearch-5.5.0.jar:5.5.0]
	at org.elasticsearch.transport.TransportService.addressesFromString(TransportService.java:691) ~[elasticsearch-5.5.0.jar:5.5.0]
	at org.elasticsearch.discovery.zen.UnicastZenPing.lambda$null$0(UnicastZenPing.java:212) ~[elasticsearch-5.5.0.jar:5.5.0]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_131]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:569) [elasticsearch-5.5.0.jar:5.5.0]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]

The discovery service seems to run:

kubectl get services
NAME                                 CLUSTER-IP   EXTERNAL-IP   PORT(S)                         AGE
kubernetes                           10.0.0.1     <none>        443/TCP                         16m
my-release-elasticsearch             10.0.0.155   <pending>     9200:30709/TCP,9300:30068/TCP   2m
my-release-elasticsearch-discovery   10.0.0.231   <none>        9300/TCP                        2m

A possibility to deploy elasticserach on dedicated nodes

** FEATURE REQUEST**

We would like to have a possibility to deploy elasticserach on dedicated nodes with tolerations and nodeselector options.