file format support

[busimus]

file format support

[busimus]

You mean you have a plain text file that was created by logging.FileHandler and you want to load it as if you saved it from cutelog?

To do that you'd have to parse the file yourself and rewrite its contents as JSON (save a log to a file from cutelog to see how it's formatted). I didn't implement this feature natively because I thought it would be painful to try to consistently parse everything correctly since everybody likes their logs formatted differently. There's no text parsing in the code so I didn't want to write it just for this, basically.

So my thinking is this: if your log files are formatted consistently, you can write a bit of code in five minutes to translate them into JSON. But if they aren't consistent, nothing I can write will help you. Although I could try to make a basic file parser to handle the simple case.

[mhanuel26]

Hi Alexander,

Thank you for your quick response,

Actually, I have parse the log file in JSON using something like the following in the formatters section of a dictConfig

"json": { "()": "pythonjsonlogger.jsonlogger.JsonFormatter", 'format': '%(asctime)s %(name)-15s %(levelname)-8s %(pathname)s:%(lineno)d %(message)s' },

then

logging.config.dictConfig(settings.config_json)

That will produce a line in a log such as

{"asctime": "2020-06-27 12:55:21,517", "name": "setup", "levelname": "INFO", "pathname": "/my/test/script.py", "lineno": 20, "message": "TEST MESSAGE"}

But also I have tried to send the log directly to cutelog after setting the Server to parse json log files, that is, by changing the "Default serialization format" under Settings from pickle to json.

but in this case I am getting a traceback at the message reception

File "home/environment/lib/python3.7/site-packages/cutelog/listener.py", line 167, in run logDict = self.deserialize(data) File "/usr/lib/python3.7/json/__init__.py", line 343, in loads s = s.decode(detect_encoding(s), 'surrogatepass')
I am interested to make it work for json either to load from a file and also to receive from a client, and I suspect cutelog is not been able to parse in any case even if both are in json.

Could you let me know how to debug please,

Many thanks,
Manuel

[busimus]

You say that you're "sending the log directly", which could mean that you're opening a socket yourself and writing an encoded JSON string to it. If that is what you're doing, the fix is simple: you need to prefix your JSON payload with the length of that payload as described on this page in the Connection section. Basically like payload = struct.pack('>L', len(json_log)) + json_log (assuming json_log is a UTF-8 encoded string, so its type is bytes).

If that's not what you're doing and instead you're trying to use the SocketHandler with the JSON formatter, then the fix is even simpler: set the default serialization format in the settings back to pickle. SocketHandler doesn't use formatters, it always sends pickled objects.

But if you're doing something else entirely, post your whole logging configuration dict and tell me how you're connecting to cutelog.

[fohrloop]

Hi,

Thanks for this awesome tool! I also am interested on how would one read records into cutelog from log files.

I managed to parse an example logfile to a list of dictionaries. Each entry in the list looks like this

{'args': None,
 'created': 1604583332.932,
 'exc_text': '',
 'filename': 'myfile.py',
 'funcName': 'myfunc',
 'levelname': 'DEBUG',
 'levelno': 10,
 'lineno': 486,
 'module': 'module',
 'msecs': 0,
 'msg': 'Test Message',
 'name': 'mypackage.myfile',
 'pathname': 'c:\\somepath\\mypackage\\myfile.py',
 'process': 1,
 'processName': 'MainProcess',
 'relativeCreated': 45525,
 'stack_info': None,
 'thread': 8568,
 'threadName': 'mythread',
 'extra_column': '',
 'datetime': datetime.datetime(2020, 11, 5, 15, 35, 32, 932000)}

How would be the preferred way to open the data in cutelog? Can I just save the list of dicts in some format and use "File -> Load Records", or should I try sending the data via sockets? Or something else?

Edit

Oh, seems that I was already really close. I just had to remove the 'datetime' entry since it cannot converted to JSON, and simply

import json
with open(outfile, "w") as f:
    json.dump(all_contents, f)

Then, the file could be loaded to cutelog with "File -> Load Records"

[SaeidJavadi]

Hi @mhanuel26 @busimus
This problem is resolved with this commit