mike-mosher/aws-la

AWS Log Analyzer -- Send AWS logs to ELK running locally in Docker containers

aws-la's Issues

VCP Flowlog seem to broken now

I am following the guide at Readme.md on MAC, but failed at import log step:

$ python importLogs.py --logtype vpc --logdir /Users/wumark/workspace/temp/vpclog/

Beginning import process
Creating mapping in ES for index: vpc_flowlogs
Creating Ingest Pipeline for index: vpc_flowlogs
Creating new index-pattern in .kibana index
Setting formatted fields on index-pattern
Setting index-pattern as default index
Deleting useless index-patterns in .kibana index
Deleting index-pattern: .ml-anomalies-*
Deleting index-pattern: .ml-notifications
importing saved objects into Kibana
Begin importing log files
File: .DS_Store is not the correct format. File need to end with *.gz
Importing log file: /Users/wumark/workspace/temp/vpclog//667164967571_vpcflowlogs_ap-northeast-1_fl-09efe29fb030b37b0_20190225T0645Z_9ba3c655.log.gz
Traceback (most recent call last):
File "importLogs.py", line 356, in
loadFiles()
File "importLogs.py", line 211, in loadFiles
processFiles(f)
File "importLogs.py", line 168, in processFiles
for i in status:
File "/usr/local/aws/lib/python2.7/site-packages/elasticsearch/helpers/init.py", line 306, in parallel_bulk
_chunk_actions(actions, chunk_size, max_chunk_bytes, client.transport.serializer)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 668, in next
raise value
elasticsearch.helpers.BulkIndexError: (u'500 document(s) failed to index.', [{u'index': {u'status': 500, u'_type': u'vpc_flowlogs', u'_index': u'vpc_flowlogs', u'error': {u'caused_by': {u'caused_by': {u'reason': u'Provided Grok expressions do not match field value: [version account-id interface-id srcaddr dstaddr sr (Skip)