objectrocket/elasticstat

Real-time performance monitoring of an Elasticsearch cluster from the command line

elasticstat's Issues

Elasticsearch 2.3 compatibility

Hello and thanks for this tool,
I'm trying it on a ES 2.3 cluster and there is an error:

./elasticstat -h opa.arte.local               
cluster                           status   shards  pri relo init unassign   pending tasks   time    
elasticsearch                     yellow        4    4    0    0        4               0   11:14:35
Traceback (most recent call last):
  File "./elasticstat", line 9, in <module>
    load_entry_point('elasticstat==1.2.0', 'console_scripts', 'elasticstat')()
  File "/home/dalexandre/.local/lib/python2.7/site-packages/elasticstat/elasticstat.py", line 501, in main
    elasticstat.print_stats()
  File "/home/dalexandre/.local/lib/python2.7/site-packages/elasticstat/elasticstat.py", line 418, in print_stats
    node_role = self.get_role(nodes_stats['nodes'][node_id]['attributes'])
KeyError: 'attributes'

The script tries to guess a node role with the key attributes here:

elasticstat/elasticstat/elasticstat.py

Line 418 in 6ec92a7

node_role = self.get_role(nodes_stats['nodes'][node_id]['attributes'])

but the /_nodes/stats API does not have this kind of information on my cluster, running 2.3.3.

The cat API display this information but it's not easy to parse: https://www.elastic.co/guide/en/elasticsearch/reference/current/cat-nodes.html

Add to circleci v2

Roles for ES5

5.0+ defines the node type in a list called roles whereas 2.x and below versions defined it in the attributes dictionary. In the current version with ES5 this gets represented as M/D and ALL instead of defining the specific role.

Make host a positional argument

At the risk of rustling @melderan's jimmies, I'm going to suggest making the host value a positional argument instead of an optional argument.

The current interface for running elasticstat (with a delay of 5s) is:

$ elasticstat -h 10.0.0.1:9200 5

I think this can be improved by instead making it:

$ elasticstat --delay 5 10.0.0.1:9200

I'm suggesting this for a few reasons:

• Prevent overloading of -h. This option is ubiquitously used for help, even in argparse.
• The delay value is an optional positional argument which would make perfect sense to have as an option (e.g. --delay) instead.
• The host "option" is going to be the most commonly set value in Elasticstat. I'd argue that this makes it the best candidate for the optional positional argument (defaulting to localhost).

ES 5 - Load Average Change

https://www.elastic.co/guide/en/elasticsearch/reference/5.x/breaking_50_index_apis.html

Traceback (most recent call last):
  File "/usr/local/bin/elasticstat", line 11, in <module>
    sys.exit(main())
  File "/root/.virtualenvs/elasticstat/lib/python2.7/site-packages/elasticstat/elasticstat.py", line 501, in main
    elasticstat.print_stats()
  File "/root/.virtualenvs/elasticstat/lib/python2.7/site-packages/elasticstat/elasticstat.py", line 434, in print_stats
    self.process_role(role, nodes_stats)
  File "/root/.virtualenvs/elasticstat/lib/python2.7/site-packages/elasticstat/elasticstat.py", line 369, in process_role
    row = self.process_node(current_role, node_id, nodes_stats['nodes'][node_id])
  File "/root/.virtualenvs/elasticstat/lib/python2.7/site-packages/elasticstat/elasticstat.py", line 340, in process_node
    node_segments.append(category_func(role, node_id, node))
  File "/root/.virtualenvs/elasticstat/lib/python2.7/site-packages/elasticstat/elasticstat.py", line 274, in process_node_os
    node_load_avg = node['os']['load_average']
KeyError: 'load_average'

List object has no attribute 'strip'

Hi, thanks for the utility! Running python version 2.7.10 I get this output when attempting to run

Traceback (most recent call last):
  File "/usr/local/bin/elasticstat", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/site-packages/elasticstat/elasticstat.py", line 501, in main
    elasticstat.print_stats()
  File "/usr/local/lib/python2.7/site-packages/elasticstat/elasticstat.py", line 401, in print_stats
    self.active_master = self.es_client.cat.master(h="id").strip() # needed to remove trailing newline
AttributeError: 'list' object has no attribute 'strip'

Printing the content of self.es_client.cat.master(h="id") gives me [{u'id': u'someRandomString'}]

Simply removing the .strip() portion of the offending line generates output which I assume would be the expected output. Happy to help by submitting a PR if you can point me in the right direction.

Thanks!

elasticstat crashes with KeyError: 'attributes'

This happens when I run this against a local two-node cluster of Elasticsearch 6.1.3:

cluster                           status   shards  pri relo init unassign   pending tasks   time    
docker-cluster                    yellow       40   35    0    0       30               0   20:09:58
Traceback (most recent call last):
  File "/usr/bin/elasticstat", line 11, in <module>
    load_entry_point('elasticstat==1.2.0', 'console_scripts', 'elasticstat')()
  File "/usr/lib/python2.7/site-packages/elasticstat/elasticstat.py", line 501, in main
    elasticstat.print_stats()
  File "/usr/lib/python2.7/site-packages/elasticstat/elasticstat.py", line 418, in print_stats
    node_role = self.get_role(nodes_stats['nodes'][node_id]['attributes'])
KeyError: 'attributes'

Add disk usage/disk space

It would be nice to gather IOPS or block reads/writes and disk space as options.

Can't install latest versions from pip

Hi there,

• The project does not seems to be up to date on pypi: https://pypi.python.org/pypi/elasticstat
• Running pip install elasticstat fetch 1.2.0
• Running pip install 'elasticstat==1.3.1' does not work
• Only solution I found was to run pip install -Iv https://github.com/objectrocket/elasticstat/archive/1.3.1.tar.gz

Thanks, 👍

Create Automated Build on Docker Hub

Docker Hub allows you to create Automated Builds from source: https://docs.docker.com/docker-hub/builds/
It would add another packaging/distribution/installation method, whose buildings would be triggered automatically on each commit. It also allows to create different image tags automatically from git tags & branches.

By making the image build via an AB, you give the resulting image verifiability and auditability. Also, the build is fully automatic. You can have the latest image tag build from HEAD and individual image tags from git's release tags. Some people avoid non-verifiable (manually uploaded) images due to security & traceability reasons.

Since #23 is in, just a free Docker Hub account and a quick setup would do. Ping me if you need help.

Elasticstat KeyError: 'http'

So after trying the version via pip 1.2.0 I ran into issue #8, so I downloaded the latest 1.3.0 and installed it, but this is the error I'm getting when I run it:

rferreira@elastic1-cab1:~/elasticstat-master$ elasticstat -h 172.24.9.71
cluster                           status   shards  pri relo init unassign   pending tasks   time
elastic-graylog                   green       110   55    0    0        0               0   10:51:25
nodes                    role                   load  mem   heap  old sz   old gc   young gc   index    search   bulk     get        fde|fdt    hconn  tconn     merges   idx st       disk usage  docs
elastic2-cab2.localnet   ALL*                   0.01  93%    13%  3.3gb    -|-      -|-        0|0|0    0|0|0    0|0|0    0|0|0        -|-        0|-     65      17.9h       0s     125.70 GB|2%  255106228
elastic1-cab2.localnet   ALL                    0.39  97%    19%  5.2gb    -|-      -|-        0|0|0    0|0|0    0|0|0    0|0|0        -|-        0|-     65      18.5h       0s     129.80 GB|2%  254607932
elastic1-cab1.localnet   ALL                    0.16  97%    66%  19.2gb   -|-      -|-        0|0|0    0|0|0    0|0|0    0|0|0        -|-        1|-     65      19.1h       0s     147.07 GB|2%  259062831
elastic2-cab1.localnet   ALL                    0.37  93%    51%  14gb     -|-      -|-        0|0|0    0|0|0    0|0|0    0|0|0        -|-        0|-     65      19.6h       0s     127.54 GB|2%  261213741
Traceback (most recent call last):
  File "/usr/local/bin/elasticstat", line 9, in <module>
    load_entry_point('elasticstat==1.3.0', 'console_scripts', 'elasticstat')()
  File "build/bdist.linux-x86_64/egg/elasticstat/elasticstat.py", line 561, in main
  File "build/bdist.linux-x86_64/egg/elasticstat/elasticstat.py", line 489, in print_stats
  File "build/bdist.linux-x86_64/egg/elasticstat/elasticstat.py", line 417, in process_role
  File "build/bdist.linux-x86_64/egg/elasticstat/elasticstat.py", line 387, in process_node
  File "build/bdist.linux-x86_64/egg/elasticstat/elasticstat.py", line 359, in process_node_connections
KeyError: 'http'

iterating trough the node keys shows that some nodes don't have that key:

thread_pool
transport_address
name
script
process
timestamp
breakers
host
fs
jvm
ip
indices
attributes
os
transport

I've made this dirty patch:

    def process_node_connections(self, role, node_id, node):
        processed_node_conns = {}
        if node.get('http') == None:
                node['http'] = {u'total_opened': 0, u'current_open': 0}
        processed_node_conns['http_conn'] = self.get_http_conns(node_id, node['http'])
        processed_node_conns['transport_conn'] = node['transport']['server_open']
        return(NODES_TEMPLATE['connections'].format(**processed_node_conns))

do you think a PR is in order?

thanks
Ricardo