airbus-cert/etl-parser

Event Trace Log file parser in pure Python

etl-parser's Issues

Question: etl2xml extracts timestamps, but in what format?

I see this timestamp in my generated xml file:

831654275173

In Windows Performance Analyzer, I see that it correlates to:
22,860,603,300 ns

I can't find any documentation anywhere that defines the timestamp format.

Any clues?

Is this library still maintained?

I just noticed it doesn't seem to be working on Windows 11, trying multiple etl files.

I also noticed this article came out around the time there stopped being activity here: https://devblogs.microsoft.com/oldnewthing/20201211-00/?p=104539

So just want to know if this is still active before I post detailed debugging info.

etl2pcap AttributeError: data1

Similar to #5

Using etl2pcap
Environment : Ubuntu 20.04
Python 3.8.2

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/construct/lib/containers.py", line 100, in __getattr__
    return self[name]
KeyError: 'data1'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/etl2pcap", line 7, in <module>
    exec(compile(f.read(), __file__, 'exec'))
  File "/home/scott_dermott/etl-parser/bin/etl2pcap", line 82, in <module>
    main(**vars(args))
  File "/home/scott_dermott/etl-parser/bin/etl2pcap", line 63, in main
    etl_reader.parse(EtlFileLogger(output_file))
  File "/home/scott_dermott/etl-parser/etl/etl.py", line 141, in parse
    actions[event.type](event.value)
  File "/home/scott_dermott/etl-parser/etl/etl.py", line 133, in <lambda>
    "EventRecord": lambda obj: observer.on_event_record(Event(obj)),
  File "/home/scott_dermott/etl-parser/bin/etl2pcap", line 34, in on_event_record
    etw = event.parse_etw()
  File "/home/scott_dermott/etl-parser/etl/event.py", line 118, in parse_etw
    guid = EtwGuid(self.source.event_header.provider_id.data1, self.source.event_header.provider_id.data2,
  File "/usr/local/lib/python3.8/dist-packages/construct/lib/containers.py", line 102, in __getattr__
    raise AttributeError(name)
AttributeError: data1

UnicodeDecodeError in Python3 installation

Just run an example using the .etl files under tests/example and using either the etl2xml or a simple script I used copying the information from the README.md. I'm receiving an utf-8 codec issue which I am not able to diagnose myself at the moment.

$ python3 test.py AMSITrace.etl 
Traceback (most recent call last):
  File "test.py", line 40, in <module>
    etl_reader = build_from_stream(etl_file.read())
  File "/usr/lib/python3.6/codecs.py", line 321, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc0 in position 75: invalid start byte

As a side note, my code is exactly the same but capturing the command line parameter:

import sys

…

if __name__ == "__main__":
    try:
        file_name = sys.argv[1]
        with open(file_name) as etl_file:
            etl_reader = build_from_stream(etl_file.read())
            etl_reader.parse(EtlFileLogger())
    except IndexError:
        print("Not enough parameters. Add the .etl file as a parameter")

File "/usr/local/lib/python3.8/dist-packages/etl/etl.py", line 159, in build_from_stream

I have installed eta-parser using the following:

git clone https://github.com/ezaspy/etl-parser
cd etl-parser
python3 -m pip install -e .

But when I run the command python3 etl-parser/bin/etl2xml -i Terminal-Services-Core.etl -o Terminal-Services-Core.etl.xml I get the following error:

Traceback (most recent call last):
  File "etl-parser/bin/etl2xml", line 252, in <module>
    main(**vars(args))
  File "etl-parser/bin/etl2xml", line 230, in main
    etl_reader = build_from_stream(input_file.read())
  File "/usr/local/lib/python3.8/dist-packages/etl/etl.py", line 159, in build_from_stream
    event_header_chunk = ChunkParser.parse(chunks[0].payload)
IndexError: list index out of range

I would expect to receive no error...

etl.error.TlUnhandledTag: Cannot read tag type 23

When trying to parse following file with etl2xml
etl.zip
I got this error

Traceback (most recent call last):
  File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python37_64\Scripts\etl2xml", line 248, in <module>
    main(**vars(args))
  File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python37_64\Scripts\etl2xml", line 227, in main
    etl_reader.parse(logger)
  File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python37_64\lib\site-packages\etl\etl.py", line 141, in parse
    actions[event.type](event.value)
  File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python37_64\lib\site-packages\etl\etl.py", line 133, in <lambda>
    "EventRecord": lambda obj: observer.on_event_record(Event(obj)),
  File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python37_64\Scripts\etl2xml", line 190, in on_event_record
    data.append(log_tracelogging(event.parse_tracelogging()))
  File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python37_64\lib\site-packages\etl\event.py", line 129, in parse_tracelogging
    return build_tracelogging(self.source)
  File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python37_64\lib\site-packages\etl\parsers\tracelogging.py", line 75, in build_tracelogging
    return TraceLogging().load(extended_data.data_item, event.user_data)
  File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python37_64\lib\site-packages\etl\parsers\tracelogging.py", line 181, in load
    self[field.name] = read_field(stream, field.tag_in)
  File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python37_64\lib\site-packages\etl\parsers\tracelogging.py", line 132, in read_field
    raise TlUnhandledTag(tag)
etl.error.TlUnhandledTag: Cannot read tag type 23

Minimal example of how I could get the traits on an event into a Panda dataframe?

I'm not extremely experienced with Python and some of the data structures involved in the way this library parses, such as constructs, leave me fairly confused about how to actually parse and iterate over the ETL file.

I noticed there are different types of events, and there seems to be some nesting involved, so maybe this question is more complicated than I realize. I've tinkered with this library for a few hours exploring the data structure of what build_from_stream returns. It seems like the properties are located within etl_reader.header.pattern.subcons but I'm having trouble reading the contents of each property.

Is it possible to show a very minimal example of how to go about converting a file into a panda dataframe?

Attribute Error data1 running etl2xml

I'm running etl2xml on this file and the event parser throws an error on all events of the file

TlMetaDataNotFound

Not sure if this is a duplicate of #5

  File "/home/USER/.local/bin/etl2xml", line 194, in on_event_record
    data.append(log_tracelogging(event.parse_tracelogging()))
  File "/home/USER/.local/lib/python3.10/site-packages/etl/event.py", line 129, in parse_tracelogging
    return build_tracelogging(self.source)
  File "/home/USER/.local/lib/python3.10/site-packages/etl/parsers/tracelogging.py", line 79, in build_tracelogging
    raise TlMetaDataNotFound()
etl.error.TlMetaDataNotFound: Meta data not found for trace logging parser

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/USER/.local/lib/python3.10/site-packages/construct/lib/containers.py", line 98, in __getattr__
    return self[name]
KeyError: 'data1'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/USER/.local/bin/etl2xml", line 252, in <module>
    main(**vars(args))
  File "/home/USER/.local/bin/etl2xml", line 231, in main
    etl_reader.parse(logger)
  File "/home/USER/.local/lib/python3.10/site-packages/etl/etl.py", line 141, in parse
    actions[event.type](event.value)
  File "/home/USER/.local/lib/python3.10/site-packages/etl/etl.py", line 133, in <lambda>
    "EventRecord": lambda obj: observer.on_event_record(Event(obj)),
  File "/home/USER/.local/bin/etl2xml", line 198, in on_event_record
    etw = event.parse_etw()
  File "/home/USER/.local/lib/python3.10/site-packages/etl/event.py", line 118, in parse_etw
    guid = EtwGuid(self.source.event_header.provider_id.data1, self.source.event_header.provider_id.data2,
  File "/home/USER/.local/lib/python3.10/site-packages/construct/lib/containers.py", line 100, in __getattr__
    raise AttributeError(name)
AttributeError: data1

Somebody send me .etl files to analyse a problem with his mail client. I'm trying to read them on my ubuntu workstation and can't convert either of them to xml.

ModuleNotFoundError: No module named 'etl'

I have installed the module 'etl' but I receive the following error when running command:

python3 etl2xml -i ../../wmi/Terminal-Services-Core.etl -o ../../wmi/Terminal-Services-Core.xml
Traceback (most recent call last):
  File "/Users/Ben/Desktop/elrond_dev/etl-parser/bin/etl2xml", line 9, in <module>
    from etl.error import GroupNotFound, VersionNotFound, EventTypeNotFound, EtwVersionNotFound, EventIdNotFound, \
ModuleNotFoundError: No module named 'etl.error'