igabriel85/dice-monitoring

IeAT Repository for H2020 DICE Project

dice-monitoring's Issues

ConnectionErrorException for parallel-ssh Issue

When a ConnectionErrorException occurs the job hangs a long time. Should be marked as failed after a certain amount of time.

Duplicate output in parallel-ssh

All functions that use parallel-ssh have duplicate output.
See ParallelSSH/parallel-ssh#27 for more details.

nmap functionality

Enhance the functionality of nmap function.

aux component deployment

Need to rewrite logstash-forwarder and collectd deployment for distributed logstash deployment

Parsing e notation values

Some values sent to logstash are in exponential notation (i.e. -1231312E2).
How to pars them and enter them into ElasticSearch.
Currently these values are defined as %{NOTSPACE}

Could use regex:

• [-+]?[0-9]*.?[0-9]+([eE][-+]?[0-9]+)?

One button bootstrapping

Implement resource - > /dmon/v1/overlord/nodes to start/install all monitoring components on registered nodes

failed to parse

1442411738648 ugi.UgiMetrics: Context=ugi, Hostname=vm-cluster-oryx3, LoginSuccessNumOps=0, LoginSuccessAvgTime=0.0, LoginFailureNumOps=0, LoginFailureAvgTime=0.0, GetGroupsNumOps=0, GetGroupsAvgTime=0.0

Add Parameter content type to swagger UI

Currently only application/json is supported as content type for payloads.

1. Fix issue by adding other content type
2. encode cert files into a json structure.

Use temp file and secure_filename

Use a temporary file for csv output of ES Core query

Use secure_filename for ssl certificate and config file uploading.

Local vs Distributed

Still need a more clear differentiation between a local deployment and a fully distributed one.

fail parse

1442491770248 ugi.UgiMetrics: Context=ugi, Hostname=vm-cluster-oryx4, LoginSuccessNumOps=0, LoginSuccessAvgTime=0.0, LoginFailureNumOps=0, LoginFailureAvgTime=0.0, GetGroupsNumOps=0, GetGroupsAvgTime=0.0

D-Mon service

How to fix redeployment issue if dmon is stopped and cannot restart because attached processes (esCore and LSCore) still running.

• If dmon fails detach esCore and LSCore

collectd template issue

collectd template generates invalid conf files.

Problem related to $end found at line 34 instead of EOL. Tried 'wb' parameter when writing files.

Editing with nano fixes issue.

Fix start.py flag issue

The -i option should work without the need to specify the port or ip

Unify naming across all json responses

Need to unify the responses for all REST API jsons.

ex:

LS Host can be Host or LSHost.

Need to create a precise naming convention.

Handle exception when trying to create duplicate DB entries

If entries have the same IP and/or FQDN sqlalchemy returns an error. Catch exception and run the app.

Restructure code

Split code into different modules:

• dbModule.py used instead of copying everything into pyDMON
• create init.py

Create minimal WUI

Create wui both:

*Bootstrap
*Eclipse plugin

rpc.rpc duplicate values

rpc.rpc context parsing produces duplicate values.
Check grok parsing.

Restructure api.models

Use inheritance for api.models for common fields such as IP and HostFQDN.

flask app starts twice

Seems that flask app starts twice in debug mode:

http://stackoverflow.com/questions/9449101/how-to-stop-flask-from-initialising-twice-in-debug-mode

Aux component selective start/stop

Need to modify resource: /v1/overlord/aux//
to => /v1/overlord/aux/deploy//

Need to add resource: /v1/overlord/aux/start//
and /v1/overlord/aux/stop//

fix virtualenv

create a viable virtualenv for D-Mon distribution and fix dmon-start.sh

Allow/Generate ssl cert

Allow/generate the uploading of ssl certificates (both keys and certs)

WUI to create elasticsearch

Create dropdown list or graph form to create the query body for elasticsearch

create log file for D-Mon

Redirect output of start.py to a log file.

Issue submitting unique values to db

If duplicate values are inserted into dmon.db it freezes in a roleback state. Catch exception and handle it gracefully.

Mapping Conflict

Some values are defined as several types (string, int, float).
Check correct type and repair index.

Revise Documentation

Revise the documentation and check for missing configuration files.

• Missing logstash.conf in Vagrant CDH Cluster setup

Ability to delete es and ls Core components.

delete ES Core and LS Core components and rebootstrap them if necessary.

Upload properties file

Create resource that uploads the propertie2 file to all nodes.
Same for Spark and Storm.

Custom ElasticSearch index

Define custom ElasticSearch index for testing end deployments.
This eliminates mapping conflicts in the future.

Graceful shutdown

Create graceful closing of application avoid:

Exception KeyError: KeyError(140674196921712,) in <module 'threading' from '/usr/lib/python2.7/threading.pyc'> ignored

Monitor Storm

Monitor storm metrics.

String concatenation

In CPython, you may want to replace:
s = head + body + maybe + tail
with the admittedly less readable:
s = "%(head)s%(body)s%(maybe)s%(tail)s" % locals()
or even:
s = "{head}{body}{maybe}{tail}".format(**locals())

queryESCore() metrics selection only for csv

Metrics selection using the arguments allm and dMetrics only works with csv output not json.

Bootstrap script pyOverloard

Create startup script similar to modaclouds-loadbalancer-controller.

Clean hostnames in collectd and ELK

Some hostnames are set in collectd by hand (i.e. DICE CDH slave4).
Remove custom name and use FQDN (i.e. dice.cdh5.s4.internal) or hostname (i.e. 109-231-126-222)

Authentification

Create authentification for overlord resource components.

Create grok patterns

Create custom patterns in grok for metrics2 file sink outputs.

For example:
%{EXPNOT}=[-+]?[0-9]*.?[0-9]+([eE][-+]?[0-9]+)? -> for exponential (scientific) notation

deployment on nginx

Deployment of flask on nginx

http://flask.pocoo.org/docs/0.10/deploying/uwsgi/

Function installLogstashForwarder fail

The function installLogstashForwarder fails to upload config file and than start logstash-forwarder service.

Export OSLC Perf Mon 2.0 RDF+XML

Export OSLC Perf Mon 2.0 RDF+XML for Collectd metrics.

http://open-services.net/wiki/performance-monitoring/OSLC-Performance-Monitoring-Specification-Version-2.0/

Monitor Spark

Monitor Spark metrics.

Document Known responses of each resource

@api.response()

http://flask-restplus.readthedocs.org/en/latest/documenting.html

support more than one Logstash Server

Extend the support for more than one LSS instance support inside DB and generate configuration for each.

Handle no metrics in defined timeframe

Exception happened during processing of request from ('79.118.13.40', 58773)
Traceback (most recent call last):
File "/usr/lib/python2.7/SocketServer.py", line 295, in _handle_request_noblock
self.process_request(request, client_address)
File "/usr/lib/python2.7/SocketServer.py", line 321, in process_request
self.finish_request(request, client_address)
File "/usr/lib/python2.7/SocketServer.py", line 334, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "/usr/lib/python2.7/SocketServer.py", line 649, in init
self.handle()
File "/usr/local/lib/python2.7/dist-packages/werkzeug/serving.py", line 200, in handle
rv = BaseHTTPRequestHandler.handle(self)
File "/usr/lib/python2.7/BaseHTTPServer.py", line 340, in handle
self.handle_one_request()
File "/usr/local/lib/python2.7/dist-packages/werkzeug/serving.py", line 235, in handle_one_request
return self.run_wsgi()
File "/usr/local/lib/python2.7/dist-packages/werkzeug/serving.py", line 177, in run_wsgi
execute(self.server.app)
File "/usr/local/lib/python2.7/dist-packages/werkzeug/serving.py", line 167, in execute
for data in application_iter:
File "/usr/local/lib/python2.7/dist-packages/werkzeug/debug/init.py", line 88, in debug_application
app_iter = self.app(environ, start_response)
File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1836, in call
return self.wsgi_app(environ, start_response)
File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1817, in wsgi_app
response = self.full_dispatch_request()
File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1475, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1461, in dispatch_request
return self.view_functionsrule.endpoint
File "/usr/local/lib/python2.7/dist-packages/flask_restful/init.py", line 471, in wrapper
resp = resource(_args, *_kwargs)
File "/usr/local/lib/python2.7/dist-packages/flask/views.py", line 84, in view
return self.dispatch_request(_args, *_kwargs)
File "/usr/local/lib/python2.7/dist-packages/flask_restful/init.py", line 581, in dispatch_request
resp = meth(_args, *_kwargs)
File "/opt/IeAT-DICE-Repository/src/pyDMON.py", line 340, in post
dict2CSV(ListMetrics,request.json['DMON']['fname'])
File "/opt/IeAT-DICE-Repository/src/pyESController.py", line 171, in dict2CSV
sys.exit("listValues argument is empty. Please supply valid input!")
SystemExit: listValues argument is empty. Please supply valid input!

Metrics selection returns complete json not just metrics specific

The values assigned to the metrics list from queryES model are not take into consideration when json type file is required.

......................................
                       metrics = request.json['DMON']['metrics']
            ListMetrics, resJson = queryESCore(query, allm=False,dMetrics=metrics, debug=False)
            #repeated from before create function
            if ftype == 'csv':
                if not 'fname' in request.json['DMON']:
                    fileName = 'output'+'.csv'
                    dict2CSV(ListMetrics)
                else:
                    fileName = request.json['DMON']['fname']+'.csv'
                    dict2CSV(ListMetrics,request.json['DMON']['fname'])
                csvOut = os.path.join(outDir,fileName)
                try:
                    csvfile=open(csvOut,'r')
                except EnvironmentError:
                    response = jsonify({'EnvError':'file not found'})
                    response.status_code = 500
                    return response
                return send_file(csvfile,mimetype = 'text/csv',as_attachment = True)
            if ftype == 'json':
                response = jsonify({'DMON':resJson})
                response.status_code = 200
                return response
......................................

Return Monitored Node

Return 2 lists instead of one:

• Running Nodes
• Stopped Nodes

Check node status function

Decide when to check node status of nodes and change nStatus value in db.