jswidler/elasticsearch-kubed

Starter files to deploy a high performance Elasticsearch cluster on Kubernetes running on either GCP or AWS

elasticsearch-kubed's Issues

config should allow user to add in name of storage class

config should allow user to add in name of storage class, in my case we want to use the ceph-block-storage .

Error Converting YAML to JSON

Hey Guys,

I am trying to install ELK in Kubernetes on a Openstack environment. I am following the instructions on the document below but I am getting the following error when I try to start elasticsearch from the template.

Doc:

https://engineering.udacity.com/high-performance-elk-with-kubernetes-part-1-1d09f41a4ce2

Error:

kubectl apply -f 2_elasticsearch
error parsing 2_elasticsearch/es-config.yml: error converting YAML to JSON: yaml: invalid map key: map[interface {}]interface {}{"namespace":interface {}(nil)}
error parsing 2_elasticsearch/es-data.yml: error converting YAML to JSON: yaml: line 140: did not find expected key
error parsing 2_elasticsearch/es-master.yml: error converting YAML to JSON: yaml: invalid map key: map[interface {}]interface {}{"namespace":interface {}(nil)}
error parsing 2_elasticsearch/es-services.yml: error converting YAML to JSON: yaml: invalid map key: map[interface {}]interface {}{"namespace":interface {}(nil)}

I had the same issue with the namespace template and I resolved it creating the namespace manually.

Any sugestion?

Thanks

Raul

config-template.py fails

I'm running python 3.6 on Ubuntu:

Python 3.6.7 (default, Oct 22 2018, 11:32:17) 
[GCC 8.2.0] on linux

and I get the following error:

config-templates.py 
These scripts will create configuration files to set up an Elasticsearch cluster in Kubernetes.
Enter a kubernetes namespace for the elasticsearch cluster (default='default'): 
Enter a name for the elasticsearch cluster (default='my-es-cluster'): 
Select the node size: 
1: minikube
2: 4cpu X N
3: 8cpu X N
4: 16cpu X N
[1-4]:  (default='2'): 2
Enter the number of nodes (2-9) (default='2'): 4
Enter the data volume size in GB [10-9999] (default='250'): 50
Using keys and certs for Logstash found in clusters/default/logstash-ssl-keys.
Would you like to configure oauth2_proxy to authorize a GitHub team? (y/N): 
Traceback (most recent call last):
  File "config-templates.py", line 185, in <module>
    main()
  File "config-templates.py", line 169, in main
    output = jinja_env.get_template(template).render(context)
  File "/home/federico/.local/lib/python3.6/site-packages/jinja2/asyncsupport.py", line 76, in render
    return original_render(self, *args, **kwargs)
  File "/home/federico/.local/lib/python3.6/site-packages/jinja2/environment.py", line 1008, in render
    return self.environment.handle_exception(exc_info, True)
  File "/home/federico/.local/lib/python3.6/site-packages/jinja2/environment.py", line 780, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/home/federico/.local/lib/python3.6/site-packages/jinja2/_compat.py", line 37, in reraise
    raise value.with_traceback(tb)
  File "templates/6_logstash/logstash-pipelines.yml", line 9, in top-level template code
    {{ inc('6_logstash/logstash.conf')|indent(first=true) }}
TypeError: do_indent() got an unexpected keyword argument 'first'

which seems to be caused by this line in the pipeline template:

elasticsearch-kubed/templates/6_logstash/logstash-pipelines.yml

Line 9 in b3446c4

{{ inc('6_logstash/logstash.conf')|indent(first=true) }}

I haven't tried upgrading to Python 3.7, that might solve the problem but ubuntu installs 3.6 by default. I'm just letting you know of the issue.

---

Just by looking at the documentation of jinja, I can see they broke backward compatibility by renaming firstIndent to first, somewhere along the release process.

Substituting first back to indentfirst solves the issue for me.

Breaking Changes on Kubernetes 1.16

I have got some Issues on 1.16

unable to recognize "5_beats_agents/filebeat-kubernetes.yml": no matches for kind "DaemonSet" in version "extensions/v1beta1"
unable to recognize "5_beats_agents/metricbeat-kubernetes.yml": no matches for kind "DaemonSet" in version "extensions/v1beta1"
unable to recognize "5_beats_agents/metricbeat-kubernetes.yml": no matches for kind "Deployment" in version "apps/v1beta1"

This one is kinda easy as you just change as per
https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.16.md#deprecations-and-removals

Will submit a PR for that

But as I am writing this there is a following error that I haven't checked yet:

error validating "5_beats_agents/filebeat-kubernetes.yml": error validating data: ValidationError(DaemonSet.spec): missing required field "selector" in io.k8s.api.apps.v1.DaemonSetSpec; if you choose to ignore these errors, turn validation off with --validate=false
error validating "5_beats_agents/metricbeat-kubernetes.yml": error validating data: ValidationError(DaemonSet.spec): missing required field "selector" in io.k8s.api.apps.v1.DaemonSetSpec; if you choose to ignore these errors, turn validation off with --validate=false

[ISSUE] Pod has unbound immediate...

OS: Centos7
Config for .yaml files:

Hi

I was reading and recreating the installation in your blog for ELK and when I execute:

The following error is showed:

I have checked my volume and it is in pending state.

Why is this happening?
How can I know the reason why the volume claim is pending?

I would appreciate any comments about it to guide me in finding the solution.

Regards.

es-config doesn't honor custom namespace

es-config uses default namespace even if custom is specified.

How to increase maxium event on logstash

Hi, I try using logstash but if icheck on elasticsearch using kibana maximum event just 600 event/second. if i try using elk non kubernetes cluster event can be 2k-5k

thanks

Logstash SSL cert not in PKCS#8 format

Note that logstash requires PKCS#8 format not PEM.

I changed the ssl_gen.sh so it creates correct format but also retains the PEM key.

function make_key_and_csr {
  openssl genrsa -out $1.pem 2048
  openssl pkcs8 -in $1.pem -topk8 -nocrypt -out $1.key
  openssl req -new -key $1.key -out $1.csr -subj "/C=$COUNTRY/ST=$STATE/L=$LOC/O=$ORG/OU=$ORG_UNIT/CN=$2"
}

Direct connect to elastic-master

Thanks a lot for the guide, very useful. I am very new with K8s and I am getting some grief from the elastic team about the fact that they can’t connect directly to the elasticsearch-master servers to import data as they do on a regular physical/virtual host architecture. I have not found a way to expose that headless service. Can anyone think on any way I can expose the elasticsearch server as we do the kibana and logstash, or if think on any way of doing that without exposing elasticsearch-master I am all ears.

Thanks in advance