json log file with an array of json log - parse

[chaudharydeepak]

json log file with an array of json log - parse

[chaudharydeepak]

I tried this way

find . -name "*.json.gz" -exec gunzip -c {} \; | jq -c '.[]' | docker run --interactive --rm --env LANG=$LANG allinurl/goaccess --date-spec=hr --real-os --agent-list --output html --log-format=--log-format='{"http_method": "%m", "request": "%U", "protocol": "%H", "response_code": "%s","referrer": "%R", "user_agent": "%u", "request_time": "%T"}' --date-format=%d/%b/%Y --time-format=%T - > report.html

This throws error:

[SETTING UP STORAGE -] {0} @ {0/s}
==1== GoAccess - version 1.7.2 - Apr 1 2023 00:44:06
==1== Config file: /etc/goaccess/goaccess.conf
==1== https://goaccess.io - [email protected]
==1== Released under the MIT License.
==1==
==1== FILE: -
==1== Parsed 10 lines producing the following errors:
==1==
==1== Token '+0000' doesn't match specifier '%m'
==1== Token '+0000' doesn't match specifier '%m'
==1== Token '+0000' doesn't match specifier '%m'
==1== Token '+0000' doesn't match specifier '%m'
==1== Token '+0000' doesn't match specifier '%m'
==1== Token '+0000' doesn't match specifier '%m'
==1== Token '+0000' doesn't match specifier '%m'
==1== Token '+0000' doesn't match specifier '%m'
==1== Token 'tRNHFlSkFyZzJoaGN2WVh6OVpXb0lnRWgvbHF0Sk54b2ZZMnQ4bWc5TER1UVhEcFgwU09DbW9tL2Jka0Mzb1JVUE5SditKVjdhS0VGaG5DODV6ZVdxcVMvWlZqcjJoZnFiT1c2TDFaaU9CNnVuRHc4eFRMZTZOWWNXWGxVT0ZkMGg0Zm54MzUyTjNoVWs2aEloOC9pVVkrUDc4QXhSTTRlTjRPYTdHdz09; _ga=GA1.2.925074475.1683821571; OptanonConsent=isGpcEnabled=0&datestamp=Mon+Oct+23+2023+15%3A59%3A20+GMT-0500+(Central+Daylight+Time)&version=202304.1.0&isIABGlobal=false&hosts=&consentId=66d79815-5717-4889-bb8f-cb19388de7da&interactionCount=1&landingPath=NotLandingPage&groups=C0001%3A1%2CC0002%3A1%2CC0003%3A1%2CC0004%3A1&AwaitingReconsent=false&geolocation=US%3BNE&browserGpcFlag=0; _ga_YGG4CDX0K1=GS1.1.1698094748.1.1.1698094760.48.0.0; AWSALB=FmgAA5D5n7cWCl5ANeBQmwMQ7L4/kMJCLHnB/ofh/m7nBLdln+1f1Onq024Q0Le9z3cpmYbe0XhrYQ0/bdGug1QlTsh30I8yMMaLFkcfVg2csIRaE/K2z2gMQc9F; AWSALBCORS=FmgAA5D5n7cWCl5ANeBQmwMQ7L4/kMJCLHnB/ofh/m7nBLdln+1f1Onq024Q0Le9z3cpmYbe0XhrYQ0/bdGug1QlTsh30I8yMMaLFkcfVg2csIRaE/K2z2gMQc9F' doesn't match specifier '%m'
==1== Token '+0000' doesn't match specifier '%m'
==1==
==1== Format Errors - Verify your log/date/time format

any pointers here pls? It seem date time format is invalid?

[chaudharydeepak]

I was able to resolve with some workarounds -

1. First I need to convert JSON multiline formatted log file to file with json payload in single lines vs multiline.
2. Then I used below log format to analyse the log-
   --log-format='{"time": "%d:%t %^", "source_ip": "%h", "source_port": "%e", "destination_ip": "%h", "destination_port": "%e", "protocol": "%e", "http_method": "%m", "request": "%U", "response_code": "%s","referrer": "%R", "user_agent": "%u", "request_time": "%T"}' --date-format=%d/%b/%Y --time-format=%T

[allinurl]

Great to hear it's up and running! By the way, just a heads up, GoAccess doesn't support JSON multiline entries. If you run into any more snags, don't hesitate to drop a few lines from your log here, and I'll give it a peek. For now, let's consider this one closed