gubser/analyzer-ecnspider1

analyzer-ecnspider1's Issues

Adapt analyzer to actually use Spark rather than just collect data from HDFS

Currently the analyzer just iterates over the raw data and applies the analysis in the local python interpreter. Instead it should run it in Spark, the thing i have to think about is what happens if a file is corrupt. Data corruption should not raise an analyzer error, instead it should notify the core that it is not happy with the data. (Also see mami-project/pto-core#20)

Example: download the ipfix file from the measurement server without having stopped QoF beforehand. the ipfix reader will raise an exception because there is no end signature(?) in the file.

remove analyzer_id from required fields in validator

The analyzer module does not necessarily know it's analyzer_id. Because multiple "versions" (i.e. branches) of the same analyzer can be in use the analyzer code most likely uses the value ac.analyzer_id provided by the supervisor anyway. Thus, the validator should take care of setting this field as it does with the others action_id & valid already.