jaegeral/redlinetimelinedb_extractor

Extracts the Timeline of a Redline parsed sqlite database so it can be used e.g. in an ELK stack.

redlinetimelinedb_extractor's Issues

Exception Thrown When Writing Unicode To CSV

Problem:

It appears that some .mans files generated by HX (from default Comprehensive Investigative Details acquisition script) contain unicode strings - I haven't managed to pinpoint which EventType(s) this applies to.

.mans file was +4.5GB in size, console output below:

2019-08-14 16:29:41,912 - rltldb_application - INFO - Redline Timeline DB <-> CSV Converter
2019-08-14 16:29:41,914 - rltldb_application - INFO - Converting file at .\REDACTED_Comprehensive Investigative Details_2019-08-10T06_04_57Z.mans
2019-08-14 16:31:08,470 - rltldb_application - INFO - Number of rows: 5975022
2019-08-14 16:31:08,472 - rltldb_application - INFO - Processing 1-10001 / 5975022
2019-08-14 16:31:08,774 - rltldb_application - INFO - Processing 10001-20001 / 5975022
2019-08-14 16:31:09,044 - rltldb_application - INFO - Processing 20001-30001 / 5975022
2019-08-14 16:31:09,322 - rltldb_application - INFO - Processing 30001-40001 / 5975022
2019-08-14 16:31:09,697 - rltldb_application - INFO - Processing 40001-50001 / 5975022
2019-08-14 16:31:10,084 - rltldb_application - INFO - Processing 50001-60001 / 5975022
2019-08-14 16:31:10,475 - rltldb_application - INFO - Processing 60001-70001 / 5975022
2019-08-14 16:31:10,778 - rltldb_application - INFO - Processing 70001-80001 / 5975022
<p>Error: <class 'UnicodeEncodeError'></p>

Solution:

Specify utf-8 encoding when opening .csv file.

Current
47: with open(csvfilename, 'w', newline='') as csvfile:
Solution
47: with open(csvfilename, 'w', newline='', encoding='utf-8') as csvfile: