jswidler / elasticsearch-kubed Goto Github PK
View Code? Open in Web Editor NEWStarter files to deploy a high performance Elasticsearch cluster on Kubernetes running on either GCP or AWS
License: MIT License
Starter files to deploy a high performance Elasticsearch cluster on Kubernetes running on either GCP or AWS
License: MIT License
config should allow user to add in name of storage class, in my case we want to use the ceph-block-storage .
Hey Guys,
I am trying to install ELK in Kubernetes on a Openstack environment. I am following the instructions on the document below but I am getting the following error when I try to start elasticsearch from the template.
Doc:
https://engineering.udacity.com/high-performance-elk-with-kubernetes-part-1-1d09f41a4ce2
Error:
kubectl apply -f 2_elasticsearch
error parsing 2_elasticsearch/es-config.yml: error converting YAML to JSON: yaml: invalid map key: map[interface {}]interface {}{"namespace":interface {}(nil)}
error parsing 2_elasticsearch/es-data.yml: error converting YAML to JSON: yaml: line 140: did not find expected key
error parsing 2_elasticsearch/es-master.yml: error converting YAML to JSON: yaml: invalid map key: map[interface {}]interface {}{"namespace":interface {}(nil)}
error parsing 2_elasticsearch/es-services.yml: error converting YAML to JSON: yaml: invalid map key: map[interface {}]interface {}{"namespace":interface {}(nil)}
I had the same issue with the namespace template and I resolved it creating the namespace manually.
Any sugestion?
Thanks
Raul
I'm running python 3.6 on Ubuntu:
Python 3.6.7 (default, Oct 22 2018, 11:32:17)
[GCC 8.2.0] on linux
and I get the following error:
config-templates.py
These scripts will create configuration files to set up an Elasticsearch cluster in Kubernetes.
Enter a kubernetes namespace for the elasticsearch cluster (default='default'):
Enter a name for the elasticsearch cluster (default='my-es-cluster'):
Select the node size:
1: minikube
2: 4cpu X N
3: 8cpu X N
4: 16cpu X N
[1-4]: (default='2'): 2
Enter the number of nodes (2-9) (default='2'): 4
Enter the data volume size in GB [10-9999] (default='250'): 50
Using keys and certs for Logstash found in clusters/default/logstash-ssl-keys.
Would you like to configure oauth2_proxy to authorize a GitHub team? (y/N):
Traceback (most recent call last):
File "config-templates.py", line 185, in <module>
main()
File "config-templates.py", line 169, in main
output = jinja_env.get_template(template).render(context)
File "/home/federico/.local/lib/python3.6/site-packages/jinja2/asyncsupport.py", line 76, in render
return original_render(self, *args, **kwargs)
File "/home/federico/.local/lib/python3.6/site-packages/jinja2/environment.py", line 1008, in render
return self.environment.handle_exception(exc_info, True)
File "/home/federico/.local/lib/python3.6/site-packages/jinja2/environment.py", line 780, in handle_exception
reraise(exc_type, exc_value, tb)
File "/home/federico/.local/lib/python3.6/site-packages/jinja2/_compat.py", line 37, in reraise
raise value.with_traceback(tb)
File "templates/6_logstash/logstash-pipelines.yml", line 9, in top-level template code
{{ inc('6_logstash/logstash.conf')|indent(first=true) }}
TypeError: do_indent() got an unexpected keyword argument 'first'
which seems to be caused by this line in the pipeline template:
I haven't tried upgrading to Python 3.7, that might solve the problem but ubuntu installs 3.6 by default. I'm just letting you know of the issue.
Just by looking at the documentation of jinja, I can see they broke backward compatibility by renaming firstIndent
to first
, somewhere along the release process.
Substituting first
back to indentfirst
solves the issue for me.
I have got some Issues on 1.16
unable to recognize "5_beats_agents/filebeat-kubernetes.yml": no matches for kind "DaemonSet" in version "extensions/v1beta1"
unable to recognize "5_beats_agents/metricbeat-kubernetes.yml": no matches for kind "DaemonSet" in version "extensions/v1beta1"
unable to recognize "5_beats_agents/metricbeat-kubernetes.yml": no matches for kind "Deployment" in version "apps/v1beta1"
This one is kinda easy as you just change as per
https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.16.md#deprecations-and-removals
Will submit a PR for that
But as I am writing this there is a following error that I haven't checked yet:
error validating "5_beats_agents/filebeat-kubernetes.yml": error validating data: ValidationError(DaemonSet.spec): missing required field "selector" in io.k8s.api.apps.v1.DaemonSetSpec; if you choose to ignore these errors, turn validation off with --validate=false
error validating "5_beats_agents/metricbeat-kubernetes.yml": error validating data: ValidationError(DaemonSet.spec): missing required field "selector" in io.k8s.api.apps.v1.DaemonSetSpec; if you choose to ignore these errors, turn validation off with --validate=false
OS: Centos7
Config for .yaml files:
Hi
I was reading and recreating the installation in your blog for ELK and when I execute:
The following error is showed:
I have checked my volume and it is in pending state.
Why is this happening?
How can I know the reason why the volume claim is pending?
I would appreciate any comments about it to guide me in finding the solution.
Regards.
es-config uses default namespace even if custom is specified.
Hi, I try using logstash but if icheck on elasticsearch using kibana maximum event just 600 event/second. if i try using elk non kubernetes cluster event can be 2k-5k
thanks
Note that logstash requires PKCS#8 format not PEM.
I changed the ssl_gen.sh so it creates correct format but also retains the PEM key.
function make_key_and_csr {
openssl genrsa -out $1.pem 2048
openssl pkcs8 -in $1.pem -topk8 -nocrypt -out $1.key
openssl req -new -key $1.key -out $1.csr -subj "/C=$COUNTRY/ST=$STATE/L=$LOC/O=$ORG/OU=$ORG_UNIT/CN=$2"
}
Thanks a lot for the guide, very useful. I am very new with K8s and I am getting some grief from the elastic team about the fact that they can’t connect directly to the elasticsearch-master servers to import data as they do on a regular physical/virtual host architecture. I have not found a way to expose that headless service. Can anyone think on any way I can expose the elasticsearch server as we do the kibana and logstash, or if think on any way of doing that without exposing elasticsearch-master I am all ears.
Thanks in advance
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.