Haproxy log format help

[truongca1902]

Haproxy log format help

[allinurl]

Is this in JSON format? Can HAProxy be configured to generate a valid JSON access log? e.g.,

log-format '{"timestamp":"%T","http_status":"%ST","frontend_name":"%f","backend_name":"%b","server_name":"%s","time_duration":"%Tc","bytes_read":"%B","captured_request_cookie":"%[capture.req.hdr(0)]","captured_response_cookie":"%[capture.res.hdr(0)]","http_request":"%r","http_referrer":"%[capture.req.hdr(Referer)]","http_user_agent":"%[capture.req.hdr(User-Agent)]","http_x_forwarded_for":"%[capture.req.hdr(X-Forwarded-For)]"}\n'

[truongca1902]

Is this in JSON format? Can HAProxy be configured to generate a valid JSON access log? e.g.,

log-format '{"timestamp":"%T","http_status":"%ST","frontend_name":"%f","backend_name":"%b","server_name":"%s","time_duration":"%Tc","bytes_read":"%B","captured_request_cookie":"%[capture.req.hdr(0)]","captured_response_cookie":"%[capture.res.hdr(0)]","http_request":"%r","http_referrer":"%[capture.req.hdr(Referer)]","http_user_agent":"%[capture.req.hdr(User-Agent)]","http_x_forwarded_for":"%[capture.req.hdr(X-Forwarded-For)]"}\n'

Yes this is JSON format,I switched to json format to push logs to another system for management
I followed the URL's instructions
https://gist.github.com/vr/c9e158e298e6e316544c399b2ff3ef22
And combined with Haproxy's instructions
https://www.haproxy.com/blog/haproxy-log-customization

This is my original log format

#log-format "%ci:%cp [%tr] %ft %b/%s %TR/%Tw/%Tc/%Tr/%Ta %ST %B %CC %CS %tsc %ac/%fc/ %bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r %[unique-id]" 

This is the log json format when pushed to another system

{ "haproxy_clientIP": "55.198.148.153", "haproxy_clientPort": "65023", "haproxy_dateTime": "04/Jan/2024:04:59:58.589", "haproxy_frontendNameTransport": "HA-webhost-950~", "haproxy_backend": "staging-exam", "haproxy_serverName": "tr1923", "haproxy_Tw": "0", "haproxy_Tc": "1", "haproxy_Tt": "+1281", "haproxy_bytesRead": "+16570", "haproxy_terminationState": "--NI", "haproxy_actconn": 7, "haproxy_FrontendCurrentConn": 5, "haproxy_backendCurrentConn": 1, "haproxy_serverConcurrentConn": 1, "haproxy_retries": 0, "haproxy_srvQueue": 0, "haproxy_backendQueue": 0, "haproxy_backendSourceIP": "192.168.1.1", "haproxy_backendSourcePort": "14523", "haproxy_statusCode": "200", "haproxy_serverIP": "192.168.1.2", "haproxy_serverPort": "443", "haproxy_frontendIP": "192.168.1.3", "haproxy_frontendPort": "443", "haproxy_capturedRequestHeaders": "{example.com|||Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/216.0.1975.83 Safari/537.36||}", "haproxy_httpRequest": "GET https://example.com/13463 HTTP/2.0", "haproxy_SSLVersion": "TLSv1.0", "haproxy_SSLCipher": "ECDHE-RSA-AES128-GCM-SHA256", "haproxy_responseTime": "616", "haproxy_capturedReponseHeaders": "{|nginx}", "haproxy_capturedRequestCookie": "-", "haproxy_capturedReponseCookie": "-" }

[allinurl]

Great, for the final sample line you provided, this solution should be effective (assuming haproxy_responseTime is measured in microseconds).

goaccess access.log --log-format='{ "haproxy_clientIP": "%h", "haproxy_dateTime": "%d:%t.%^", "haproxy_serverName": "%v", "haproxy_bytesRead": "+%b", "haproxy_statusCode": "%s", "haproxy_capturedRequestHeaders": "%u", "haproxy_httpRequest": "%r", "haproxy_SSLVersion": "%K", "haproxy_SSLCipher": "%k", "haproxy_responseTime": "%D" }' --date-format=%d/%b/%Y --time-format=%T --date-spec=min

[truongca1902]

Great, for the final sample line you provided, this solution should be effective (assuming haproxy_responseTime is measured in microseconds).

goaccess access.log --log-format='{ "haproxy_clientIP": "%h", "haproxy_dateTime": "%d:%t.%^", "haproxy_serverName": "%v", "haproxy_bytesRead": "+%b", "haproxy_statusCode": "%s", "haproxy_capturedRequestHeaders": "%u", "haproxy_httpRequest": "%r", "haproxy_SSLVersion": "%K", "haproxy_SSLCipher": "%k", "haproxy_responseTime": "%D" }' --date-format=%d/%b/%Y --time-format=%T --date-spec=min

Great, for the final sample line you provided, this solution should be effective (assuming haproxy_responseTime is measured in microseconds).

goaccess access.log --log-format='{ "haproxy_clientIP": "%h", "haproxy_dateTime": "%d:%t.%^", "haproxy_serverName": "%v", "haproxy_bytesRead": "+%b", "haproxy_statusCode": "%s", "haproxy_capturedRequestHeaders": "%u", "haproxy_httpRequest": "%r", "haproxy_SSLVersion": "%K", "haproxy_SSLCipher": "%k", "haproxy_responseTime": "%D" }' --date-format=%d/%b/%Y --time-format=%T --date-spec=min

Thanks for your help but when I run it it gives the following error

==2664179== IPv4/6 is required.
==2664179== IPv4/6 is required.
==2664179== IPv4/6 is required.
==2664179== IPv4/6 is required.
==2664179== IPv4/6 is required.
==2664179== IPv4/6 is required.
==2664179== IPv4/6 is required.
==2664179== IPv4/6 is required.
==2664179== IPv4/6 is required.
==2664179== IPv4/6 is required.

I thought my log didn't contain an ip, but when I tried creating a sample log, it still gave me the same error

[allinurl]

Please feel free to drop the log here, or share a substantial sample instead of just a single line. Thanks

[truongca1902]

Please feel free to drop the log here, or share a substantial sample instead of just a single line. Thanks

This is a standard log format in the file, all logs have this format
024-01-03T21:59:59.850104+00:00 0 2024-01-04T04: 59:59.871596+07:00 staging-ha-vm546 hapr-webhost 39034 - - {"haproxy_clientIP":"55.198.148.153","haproxy_clientPort":"65023","haproxy_dateTime":"04/Jan/2024:04:59:58.589","haproxy_frontendNameTransport":"HA-webhost-950~","haproxy_backend":"staging-exam","haproxy_serverName":"tr1923","haproxy_Tw":"0","haproxy_Tc":"1","haproxy_Tt":"+1281","haproxy_bytesRead":"+16570","haproxy_terminationState":"--NI","haproxy_actconn":7,"haproxy_FrontendCurrentConn":5,"haproxy_backendCurrentConn":1,"haproxy_serverConcurrentConn":1,"haproxy_retries":0,"haproxy_srvQueue":0,"haproxy_backendQueue":0,"haproxy_backendSourceIP":"192.168.1.1","haproxy_backendSourcePort":"14523","haproxy_statusCode":"200","haproxy_serverIP":"192.168.1.2","haproxy_serverPort":"443","haproxy_frontendIP":"192.168.1.3","haproxy_frontendPort":"443","haproxy_capturedRequestHeaders":"{example.com|||Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/216.0.1975.83 Safari/537.36||}","haproxy_httpRequest":"GET https://example.com/13463 HTTP/2.0","haproxy_SSLVersion":"TLSv1.0","haproxy_SSLCipher":"ECDHE-RSA-AES128-GCM-SHA256","haproxy_responseTime":"616","haproxy_capturedReponseHeaders":"{|nginx}","haproxy_capturedRequestCookie":"-","haproxy_capturedReponseCookie":"-"}$

[truongca1902]

I found a workaround for this using grep to only get values ​​in "{"
grep -ro {.*} access.log | goaccess --log-format='{ "haproxy_clientIP":"%h", "haproxy_dateTime": "%d:%t.%^", "haproxy_serverName": "%v", "haproxy_bytesRead": "+%b", "haproxy_statusCode": "%s", "haproxy_capturedRequestHeaders": "%u", "haproxy_httpRequest": "%r", "haproxy_SSLVersion": "%K", "haproxy_SSLCipher": "%k", "haproxy_responseTime": "%D" }' --date-format=%d/%b/%Y --time-format=%T --date-spec=min -a -o report.html

Thank you for your support

[allinurl]

Great to hear it worked! Closing for now, let me know if you have any other questions.