igabriel85 / dice-monitoring Goto Github PK
View Code? Open in Web Editor NEWIeAT Repository for H2020 DICE Project
Home Page: http://www.dice-h2020.eu/
License: Apache License 2.0
IeAT Repository for H2020 DICE Project
Home Page: http://www.dice-h2020.eu/
License: Apache License 2.0
When a ConnectionErrorException occurs the job hangs a long time. Should be marked as failed after a certain amount of time.
All functions that use parallel-ssh have duplicate output.
See ParallelSSH/parallel-ssh#27 for more details.
Enhance the functionality of nmap function.
Need to rewrite logstash-forwarder and collectd deployment for distributed logstash deployment
Some values sent to logstash are in exponential notation (i.e. -1231312E2).
How to pars them and enter them into ElasticSearch.
Currently these values are defined as %{NOTSPACE}
Could use regex:
Implement resource - > /dmon/v1/overlord/nodes to start/install all monitoring components on registered nodes
1442411738648 ugi.UgiMetrics: Context=ugi, Hostname=vm-cluster-oryx3, LoginSuccessNumOps=0, LoginSuccessAvgTime=0.0, LoginFailureNumOps=0, LoginFailureAvgTime=0.0, GetGroupsNumOps=0, GetGroupsAvgTime=0.0
Currently only application/json is supported as content type for payloads.
Use a temporary file for csv output of ES Core query
Use secure_filename for ssl certificate and config file uploading.
Still need a more clear differentiation between a local deployment and a fully distributed one.
1442491770248 ugi.UgiMetrics: Context=ugi, Hostname=vm-cluster-oryx4, LoginSuccessNumOps=0, LoginSuccessAvgTime=0.0, LoginFailureNumOps=0, LoginFailureAvgTime=0.0, GetGroupsNumOps=0, GetGroupsAvgTime=0.0
How to fix redeployment issue if dmon is stopped and cannot restart because attached processes (esCore and LSCore) still running.
collectd template generates invalid conf files.
Problem related to $end found at line 34 instead of EOL. Tried 'wb' parameter when writing files.
Editing with nano fixes issue.
The -i option should work without the need to specify the port or ip
Need to unify the responses for all REST API jsons.
ex:
LS Host can be Host or LSHost.
Need to create a precise naming convention.
If entries have the same IP and/or FQDN sqlalchemy returns an error. Catch exception and run the app.
Split code into different modules:
Create wui both:
*Bootstrap
*Eclipse plugin
rpc.rpc context parsing produces duplicate values.
Check grok parsing.
Use inheritance for api.models for common fields such as IP and HostFQDN.
Seems that flask app starts twice in debug mode:
http://stackoverflow.com/questions/9449101/how-to-stop-flask-from-initialising-twice-in-debug-mode
Need to modify resource: /v1/overlord/aux//
to => /v1/overlord/aux/deploy//
Need to add resource: /v1/overlord/aux/start//
and /v1/overlord/aux/stop//
create a viable virtualenv for D-Mon distribution and fix dmon-start.sh
Allow/generate the uploading of ssl certificates (both keys and certs)
Create dropdown list or graph form to create the query body for elasticsearch
Redirect output of start.py to a log file.
If duplicate values are inserted into dmon.db it freezes in a roleback state. Catch exception and handle it gracefully.
Some values are defined as several types (string, int, float).
Check correct type and repair index.
Revise the documentation and check for missing configuration files.
delete ES Core and LS Core components and rebootstrap them if necessary.
Create resource that uploads the propertie2 file to all nodes.
Same for Spark and Storm.
Define custom ElasticSearch index for testing end deployments.
This eliminates mapping conflicts in the future.
Create graceful closing of application avoid:
Exception KeyError: KeyError(140674196921712,) in <module 'threading' from '/usr/lib/python2.7/threading.pyc'> ignored
Monitor storm metrics.
In CPython, you may want to replace:
s = head + body + maybe + tail
with the admittedly less readable:
s = "%(head)s%(body)s%(maybe)s%(tail)s" % locals()
or even:
s = "{head}{body}{maybe}{tail}".format(**locals())
Metrics selection using the arguments allm and dMetrics only works with csv output not json.
Create startup script similar to modaclouds-loadbalancer-controller.
Some hostnames are set in collectd by hand (i.e. DICE CDH slave4).
Remove custom name and use FQDN (i.e. dice.cdh5.s4.internal) or hostname (i.e. 109-231-126-222)
Create authentification for overlord resource components.
Create custom patterns in grok for metrics2 file sink outputs.
For example:
%{EXPNOT}=[-+]?[0-9]*.?[0-9]+([eE][-+]?[0-9]+)? -> for exponential (scientific) notation
Deployment of flask on nginx
The function installLogstashForwarder fails to upload config file and than start logstash-forwarder service.
Export OSLC Perf Mon 2.0 RDF+XML for Collectd metrics.
Monitor Spark metrics.
@api.response()
http://flask-restplus.readthedocs.org/en/latest/documenting.html
Extend the support for more than one LSS instance support inside DB and generate configuration for each.
Exception happened during processing of request from ('79.118.13.40', 58773)
Traceback (most recent call last):
File "/usr/lib/python2.7/SocketServer.py", line 295, in _handle_request_noblock
self.process_request(request, client_address)
File "/usr/lib/python2.7/SocketServer.py", line 321, in process_request
self.finish_request(request, client_address)
File "/usr/lib/python2.7/SocketServer.py", line 334, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "/usr/lib/python2.7/SocketServer.py", line 649, in init
self.handle()
File "/usr/local/lib/python2.7/dist-packages/werkzeug/serving.py", line 200, in handle
rv = BaseHTTPRequestHandler.handle(self)
File "/usr/lib/python2.7/BaseHTTPServer.py", line 340, in handle
self.handle_one_request()
File "/usr/local/lib/python2.7/dist-packages/werkzeug/serving.py", line 235, in handle_one_request
return self.run_wsgi()
File "/usr/local/lib/python2.7/dist-packages/werkzeug/serving.py", line 177, in run_wsgi
execute(self.server.app)
File "/usr/local/lib/python2.7/dist-packages/werkzeug/serving.py", line 167, in execute
for data in application_iter:
File "/usr/local/lib/python2.7/dist-packages/werkzeug/debug/init.py", line 88, in debug_application
app_iter = self.app(environ, start_response)
File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1836, in call
return self.wsgi_app(environ, start_response)
File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1817, in wsgi_app
response = self.full_dispatch_request()
File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1475, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1461, in dispatch_request
return self.view_functionsrule.endpoint
File "/usr/local/lib/python2.7/dist-packages/flask_restful/init.py", line 471, in wrapper
resp = resource(_args, *_kwargs)
File "/usr/local/lib/python2.7/dist-packages/flask/views.py", line 84, in view
return self.dispatch_request(_args, *_kwargs)
File "/usr/local/lib/python2.7/dist-packages/flask_restful/init.py", line 581, in dispatch_request
resp = meth(_args, *_kwargs)
File "/opt/IeAT-DICE-Repository/src/pyDMON.py", line 340, in post
dict2CSV(ListMetrics,request.json['DMON']['fname'])
File "/opt/IeAT-DICE-Repository/src/pyESController.py", line 171, in dict2CSV
sys.exit("listValues argument is empty. Please supply valid input!")
SystemExit: listValues argument is empty. Please supply valid input!
The values assigned to the metrics list from queryES model are not take into consideration when json type file is required.
......................................
metrics = request.json['DMON']['metrics']
ListMetrics, resJson = queryESCore(query, allm=False,dMetrics=metrics, debug=False)
#repeated from before create function
if ftype == 'csv':
if not 'fname' in request.json['DMON']:
fileName = 'output'+'.csv'
dict2CSV(ListMetrics)
else:
fileName = request.json['DMON']['fname']+'.csv'
dict2CSV(ListMetrics,request.json['DMON']['fname'])
csvOut = os.path.join(outDir,fileName)
try:
csvfile=open(csvOut,'r')
except EnvironmentError:
response = jsonify({'EnvError':'file not found'})
response.status_code = 500
return response
return send_file(csvfile,mimetype = 'text/csv',as_attachment = True)
if ftype == 'json':
response = jsonify({'DMON':resJson})
response.status_code = 200
return response
......................................
Return 2 lists instead of one:
Decide when to check node status of nodes and change nStatus value in db.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.