traackr/ansible-elasticsearch

Ansible playbook for Elasticsearch

ansible-elasticsearch's Issues

vagrant up ERROR: provided hosts list is empty

vagrant up
Bringing machine 'default' up with 'virtualbox' provider...
==> default: Box 'precise64' could not be found. Attempting to find and install...
default: Box Provider: virtualbox
default: Box Version: >= 0
==> default: Adding box 'precise64' (v0) for provider: virtualbox
default: Downloading: http://files.vagrantup.com/precise64.box
==> default: Successfully added box 'precise64' (v0) for 'virtualbox'!
==> default: Importing base box 'precise64'...
==> default: Matching MAC address for NAT networking...
==> default: Setting the name of the VM: ansible-elasticsearch_default_1396965092203_15519
==> default: Fixed port collision for 22 => 2222. Now on port 2200.
==> default: Clearing any previously set network interfaces...
==> default: Preparing network interfaces based on configuration...
default: Adapter 1: nat
default: Adapter 2: hostonly
==> default: Forwarding ports...
default: 22 => 2200 (adapter 1)
==> default: Booting VM...
==> default: Waiting for machine to boot. This may take a few minutes...
default: SSH address: 127.0.0.1:2200
default: SSH username: vagrant
default: SSH auth method: private key
default: Warning: Remote connection disconnect. Retrying...
default: Warning: Remote connection disconnect. Retrying...
==> default: Machine booted and ready!
==> default: Checking for guest additions in VM...
default: The guest additions on this VM do not match the installed version of
default: VirtualBox! In most cases this is fine, but in rare cases it can
default: prevent things such as shared folders from working properly. If you see
default: shared folder errors, please make sure the guest additions within the
default: virtual machine match the version of VirtualBox you have installed on
default: your host and reload your VM.
default:
default: Guest Additions Version: 4.2.0
default: VirtualBox Version: 4.3
==> default: Configuring and enabling network interfaces...
==> default: Mounting shared folders...
default: /vagrant => /Users/miguel/Apps/ansible-elasticsearch
==> default: Running provisioner: ansible...
ERROR: provided hosts list is empty
Ansible failed to complete successfully. Any error output should be
visible above. Please fix these errors and try again.

Do not reinstall on every run

Would it be possible not to force remove and reinstall es on every run?

I think checking for version number change could be enough.

ref https://github.com/Traackr/ansible-elasticsearch/blob/master/tasks/main.yml#L39

Issue starting ES service with 2.3

Recently upgraded an ES instance from 1.7 to 2.3 and there was an issue starting the systemv service:

server:~$ sudo start-stop-daemon -d /usr/share/elasticsearch --start --user elasticsearch -c elasticsearch --pidfile /var/run/elasticsearch.pid --exec /usr/share/elasticsearch/bin/elasticsearch -- -d -p /var/run/elasticsearch.pid --default.path.home=/usr/share/elasticsearch --default.path.logs=/var/log/elasticsearch --default.path.data=/var/lib/elasticsearch --default.path.conf=/etc/elasticsearch -Des.max-open-files=true

server:~$ ERROR: Parameter [-Des.max-open-files=true] starting with "-D", "-d" or "-p" must be before any parameters starting with --

I believe this is due to the positioning of the es.max-open-files Java property on https://github.com/Traackr/ansible-elasticsearch/blob/master/tasks/elastic-install.yml#L136

To temporarily fix this, i removed the DAEMON_OPTS="$DAEMON_OPTS -Des.max-open-files=true" line from /etc/init.d/elasticsearch and added the ES_JAVA_OPTS Java opt on /etc/default/elasticsearch:

cat /etc/default/elasticsearch
# Run ElasticSearch as this user ID and group ID
#ES_USER=elasticsearch
ES_USER=elasticsearch
#ES_GROUP=elasticsearch
ES_GROUP=elasticsearch

# Heap Size (defaults to 256m min, 1g max)
#ES_HEAP_SIZE=2g

# Heap new generation
#ES_HEAP_NEWSIZE=

# max direct memory
#ES_DIRECT_SIZE=

# Maximum number of open files, defaults to 65535.
#MAX_OPEN_FILES=65535
MAX_OPEN_FILES=65535

# Maximum locked memory size. Set to "unlimited" if you use the
# bootstrap.mlockall option in elasticsearch.yml. You must also set
# ES_HEAP_SIZE.
#MAX_LOCKED_MEMORY=unlimited

# ElasticSearch log directory
#LOG_DIR=/var/log/elasticsearch

# ElasticSearch data directory
#DATA_DIR=/var/lib/elasticsearch

# ElasticSearch work directory
#WORK_DIR=/tmp/elasticsearch

# ElasticSearch configuration directory
#CONF_DIR=/etc/elasticsearch

# ElasticSearch configuration file (elasticsearch.yml)
#CONF_FILE=/etc/elasticsearch/elasticsearch.yml

# Additional Java OPTS
ES_JAVA_OPTS=-Des.max-open-files=true

# Environment Vars

PID_DIR=/var/run

Is changing /etc/defaults/elasticsearch a better option than the service file ?

Running ES process info shows property set:

jinfo 15063
Attaching to process ID 15063, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 24.80-b11
Java System Properties:

jna.platform.library.path = /usr/lib/x86_64-linux-gnu:/lib/x86_64-linux-gnu:/lib64:/usr/lib:/lib
java.runtime.name = Java(TM) SE Runtime Environment
es.default.path.logs = /var/log/elasticsearch
sun.boot.library.path = /usr/lib/jvm/java-7-oracle/jre/lib/amd64
java.vm.version = 24.80-b11
es.path.home = /usr/share/elasticsearch
es.pidfile = /var/run/elasticsearch.pid
java.vm.vendor = Oracle Corporation
java.vendor.url = http://java.oracle.com/
path.separator = :
file.encoding.pkg = sun.io
java.vm.name = Java HotSpot(TM) 64-Bit Server VM
sun.java.launcher = SUN_STANDARD
user.country = US
sun.os.patch.level = unknown
jna.nosys = true
java.vm.specification.name = Java Virtual Machine Specification
user.dir = /usr/share/elasticsearch
java.runtime.version = 1.7.0_80-b15
es.default.path.conf = /etc/elasticsearch
java.awt.graphicsenv = sun.awt.X11GraphicsEnvironment
java.endorsed.dirs = /usr/lib/jvm/java-7-oracle/jre/lib/endorsed
os.arch = amd64
java.io.tmpdir = /tmp
line.separator =

java.vm.specification.vendor = Oracle Corporation
os.name = Linux
es.default.path.data = /var/lib/elasticsearch
sun.jnu.encoding = UTF-8
jnidispatch.path = /tmp/jna--1985354563/jna8084413846082839662.tmp
java.library.path = /usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
sun.nio.ch.bugLevel =
java.specification.name = Java Platform API Specification
java.class.version = 51.0
sun.management.compiler = HotSpot 64-Bit Tiered Compilers
es.foreground = false
os.version = 3.13.0-88-generic
user.home = /home/elasticsearch
user.timezone = Etc/UTC
java.awt.printerjob = sun.print.PSPrinterJob
file.encoding = UTF-8
java.specification.version = 1.7
es.default.path.home = /usr/share/elasticsearch
es.logger.prefix =
user.name = elasticsearch
java.class.path = /usr/share/elasticsearch/lib/elasticsearch-2.3.3.jar:/usr/share/elasticsearch/lib/lucene-analyzers-common-5.5.0.jar:/usr/share/elasticsearch/lib/hppc-0.7.1.jar:/usr/share/elasticsearch/lib/jts-1.13.jar:/usr/share/elasticsearch/lib/joda-convert-1.2.jar:/usr/share/elasticsearch/lib/lucene-grouping-5.5.0.jar:/usr/share/elasticsearch/lib/jackson-dataformat-yaml-2.6.6.jar:/usr/share/elasticsearch/lib/spatial4j-0.5.jar:/usr/share/elasticsearch/lib/lucene-misc-5.5.0.jar:/usr/share/elasticsearch/lib/jsr166e-1.1.0.jar:/usr/share/elasticsearch/lib/joda-time-2.8.2.jar:/usr/share/elasticsearch/lib/lucene-backward-codecs-5.5.0.jar:/usr/share/elasticsearch/lib/compiler-0.8.13.jar:/usr/share/elasticsearch/lib/jackson-core-2.6.6.jar:/usr/share/elasticsearch/lib/lucene-core-5.5.0.jar:/usr/share/elasticsearch/lib/log4j-1.2.17.jar:/usr/share/elasticsearch/lib/jackson-dataformat-smile-2.6.6.jar:/usr/share/elasticsearch/lib/apache-log4j-extras-1.2.17.jar:/usr/share/elasticsearch/lib/lucene-queries-5.5.0.jar:/usr/share/elasticsearch/lib/lucene-memory-5.5.0.jar:/usr/share/elasticsearch/lib/guava-18.0.jar:/usr/share/elasticsearch/lib/lucene-highlighter-5.5.0.jar:/usr/share/elasticsearch/lib/lucene-suggest-5.5.0.jar:/usr/share/elasticsearch/lib/securesm-1.0.jar:/usr/share/elasticsearch/lib/netty-3.10.5.Final.jar:/usr/share/elasticsearch/lib/jackson-dataformat-cbor-2.6.6.jar:/usr/share/elasticsearch/lib/lucene-spatial-5.5.0.jar:/usr/share/elasticsearch/lib/jna-4.1.0.jar:/usr/share/elasticsearch/lib/elasticsearch-2.3.3.jar:/usr/share/elasticsearch/lib/compress-lzf-1.0.2.jar:/usr/share/elasticsearch/lib/t-digest-3.0.jar:/usr/share/elasticsearch/lib/lucene-join-5.5.0.jar:/usr/share/elasticsearch/lib/commons-cli-1.3.1.jar:/usr/share/elasticsearch/lib/HdrHistogram-2.1.6.jar:/usr/share/elasticsearch/lib/lucene-queryparser-5.5.0.jar:/usr/share/elasticsearch/lib/lucene-sandbox-5.5.0.jar:/usr/share/elasticsearch/lib/lucene-spatial3d-5.5.0.jar:/usr/share/elasticsearch/lib/snakeyaml-1.15.jar
java.vm.specification.version = 1.7
sun.arch.data.model = 64
java.home = /usr/lib/jvm/java-7-oracle/jre
es.max-open-files = true
sun.java.command = org.elasticsearch.bootstrap.Elasticsearch start -d -p /var/run/elasticsearch.pid --default.path.home=/usr/share/elasticsearch --default.path.logs=/var/log/elasticsearch --default.path.data=/var/lib/elasticsearch --default.path.conf=/etc/elasticsearch
user.language = en
java.specification.vendor = Oracle Corporation
awt.toolkit = sun.awt.X11.XToolkit
java.vm.info = mixed mode
java.version = 1.7.0_80
java.ext.dirs = /usr/lib/jvm/java-7-oracle/jre/lib/ext:/usr/java/packages/lib/ext
sun.boot.class.path = /usr/lib/jvm/java-7-oracle/jre/lib/resources.jar:/usr/lib/jvm/java-7-oracle/jre/lib/rt.jar:/usr/lib/jvm/java-7-oracle/jre/lib/sunrsasign.jar:/usr/lib/jvm/java-7-oracle/jre/lib/jsse.jar:/usr/lib/jvm/java-7-oracle/jre/lib/jce.jar:/usr/lib/jvm/java-7-oracle/jre/lib/charsets.jar:/usr/lib/jvm/java-7-oracle/jre/lib/jfr.jar:/usr/lib/jvm/java-7-oracle/jre/classes
java.vendor = Oracle Corporation
java.awt.headless = true
file.separator = /
java.vendor.url.bug = http://bugreport.sun.com/bugreport/
sun.io.unicode.encoding = UnicodeLittle
sun.cpu.endian = little
sun.cpu.isalist =

VM Flags:

-Xms256m -Xmx1g -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -Dfile.encoding=UTF-8 -Djna.nosys=true -Des.max-open-files=true -Des.path.home=/usr/share/elasticsearch

Permission Errors

I'm trying to run this role on an ubuntu server, logged in as the ubuntu user. I was receiving many permission issues, and needed to add sudo to several of the commands.

What environment are you running this role in, would it make sense for me to submit my patch upstream?

yum support

Need to adapt this to our RedHat-based systems - any interest in making this work across repositories? Potentially, I could abstract it to use either instead of just replacing all the apt parts.

Nodes don't restart on reboot

We need to execute this on on install (idempotent)

sudo /usr/sbin/update-rc.d elasticsearch defaults

... but I'm not sure how to express this in ansible...

ubuntu 11.04 init script not working

When i tried to start the elastic search using init script its not working. However it works fine when i manually run from bin/elasticsearch

idempotency & Coding style

Hi,

Thank you for sharing your code. We use it heavily. I have two suggestion.
1- idempotency
We try to use idempotent roles only
2- Coding style
Lines are too long to read and arguments in same line make it hard.

We are working on our own fork and would like to ask would you like to have a pull request with the above ?

Elasticsearch not starting

When deploying Elasticsearch with this role, Elasticsearch is not starting properly and I can find no logs. /var/logs/elasticsearch is empty. "$ sudo service elasticsearch start" says that the service has started, but then if I do a "$ sudo service elasticsearch status" it says that it is stopped.

This is my play:

---
- name: deploy elasticsearch
  hosts: all
  sudo: true
  roles:
  - common
  - gpstathis.elasticsearch
  vars_files:
  - vars/elasticsearch.yaml
  - vars/main.yaml

and my vars:

elasticsearch_cluster_name: "{{es_cluster}}"
elasticsearch_plugin_aws_ec2_groups: "{{stack}}-elasticsearch-ASG-sg"
elasticsearch_plugin_aws_ec2_ping_timeout: 3s
elasticsearch_network_http_port: 9200
elasticsearch_network_transport_tcp_port: 9300
elasticsearch_plugin_aws_access_key: "null"
elasticsearch_plugin_aws_secret_key: "null"
elasticsearch_version: "{{es_version}}"
elasticsearch_plugin_aws_version: "{{es_aws_version}}"
elasticsearch_heap_size: 1g
elasticsearch_max_open_files: 65535
elasticsearch_max_locked_memory: unlimited
elasticsearch_timezone: "America/New_York"
elasticsearch_node_max_local_storage_nodes: 1
elasticsearch_index_mapper_dynamic: "true"
elasticsearch_memory_bootstrap_mlockall: "true"
elasticsearch_gateway_type: local
elasticsearch_gateway_recover_after_nodes: 1
elasticsearch_gateway_recover_after_time: 1m
elasticsearch_gateway_expected_nodes: 3
elasticsearch_discovery_zen_minimum_master_nodes: 1
elasticsearch_discovery_zen_ping_multicast_enabled: "false"

elasticseasrch_plugins: 
- {name: 'mobz/elasticseasrch-head'}

"Install python-software-properties" is really slow

Hello.

I'm using your playbook, and I notice Install python-software-properties is very slow.

More over this task is always in a changed state (like many others BTW)

changed: [default]
 _______________________________________
/ TASK: ansible-elasticsearch | Install \
\ python-software-properties            /
 ---------------------------------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

Can we do something about that ?

refs #3

Test issue

Testing webhooks

Submit this playbook to Ansible Galaxy

This is a very nice playbook, please submit it to https://galaxy.ansible.com/ to make it more popular, it deserves it.

Support for elasticsearch 2.0

Great role. Do you plan on supporting elasticsearch 2.0 too?

Link to the galaxy

https://galaxy.ansible.com/list#/roles/181

It took way too long for me to figure out how to install this via ansible galaxy: ansible-galaxy install gpstathis.elasticsearch

THANK YOU

I'm getting ready to try deploying with this playbook in the next few minutes. Thanks a lot for providing it!

Remove code to set timezone?

Just a suggestion: this role (ansible-elasticsearch) sets the server timezone.

There are already a number of roles in Ansible Galaxy that perform this task (e.g. https://galaxy.ansible.com/Stouts/timezone/) so maybe setting the timezone could be left to another role?
It would help simplify the codebase.

PS Great work on writing this role BTW - it saved me a lot of effort!

dpkg-reconfigure must be run as root when setting timezone

When running on this ubuntu image: config.vm.box_url = "http://cloud-images.ubuntu.com/vagrant/raring/current/raring-server-cloudimg-i386-vagrant-disk1.box" :

failed: [default] => {"changed": true, "cmd": "echo Etc/UTC > /etc/timezone; dpkg-reconfigure --frontend noninteractive tzdata ", "delta": "0:00:00.045878", "end": "2014-06-09 15:17:04.249089", "item": "", "rc": 1, "start": "2014-06-09 15:17:04.203211"}
stderr: /bin/sh: 1: cannot create /etc/timezone: Permission denied
/usr/sbin/dpkg-reconfigure must be run as root

Wrong PID path on Debian

while running this playbook on Debian, the PID path is wrong. it is set to /var/run, which will require to change permissions on /var/run. The Elasticsearch packages sets up /var/run/elasticsearch/ with the correct permissions.

I've changed it in a fork, but I'm not sure it'll work on targeted distributions (I think this playbook was targeted for Ubuntu?)

Update Repositories task fails on Ansible 1.8 dev branch

TASK: [elasticsearch | Update repositories] *********************************** 
failed: [logging] => (item=ppa:webupd8team/java) => {"failed": true, "item": "ppa:webupd8team/java"}
msg: Invalid repository string: ppa:webupd8team/java

FATAL: all hosts have already failed -- aborting

PLAY RECAP ******************************************************************** 
           to retry, use: --limit @/home/dvaida/logging.retry

logging              : ok=1    changed=0    unreachable=0    failed=1  

false rewritten as False in elasticsearch.yml

a little gotcha in the elasticsearch.yml configuration. In your examples you always have "true" and "false" in quotes, but should you happen to put true and false without quotes, it ends up in the elasticsearch.yml file as False with a capital first letter, and elasticsearch doesn't understand that.