vy / log4j2-logstash-layout Goto Github PK
View Code? Open in Web Editor NEWLog4j 2.x plugin for customizable and Logstash-friendly JSON layout.
License: Apache License 2.0
log4j2-logstash-layout's Issues
How to include spring properties
i tried something like this but no luck
"env": "${spring.profiles.active}"
Support user provided ObjectMapper
Triggered by a log4j-user discussion.
Pretty print when log level is error
Well getting used to json log but it is really hard to read logs when there is error. Even though we are shipping it elastic search but it will be nice to have pretty print when the log level is error.
Support PatternLayout while rendering message
Ala Log4j 2 GelfLayout, introduce messagePattern configuration to support PatternLayout while rendering messages.
Any template ready for GCP?
I was wondering if there is any template available for usage with GCP, according to https://cloud.google.com/logging/docs/agent/configuration#process-payload
Thanks!
Custom user fields support
In the original logstash layout there is a possibility to use custom user fields:
https://github.com/logstash/log4j-jsonevent-layout#custom-user-fields
This feature is quite handy to provide, for example, a service name through log4j config.
I suppose one can do it with this lib, but It would require to use custom layout for each service which adds extra field with different value for each service. (Or maybe I didn't understand the docs). This way is not very convenient.
extract certain MDC fields
instead of having { "mdc" : { "customThreadContextField" : "someValue1", "secondCustomThreadContextField" : "someValue2", }, "@version" : 1, "source_host" : "raspberrypi", "message" : "some msg", }
I would like to have the mdc flat to the rest of the fields { "customThreadContextField" : "someValue1", "secondCustomThreadContextField" : "someValue2", "@version" : 1, "source_host" : "raspberrypi", "message" : "some msg", }
Is that somehow possible? I tried adding "customThreadContextField": "${mdc:customThreadContextField}", to a custom layout template, but the syntax is obviously wrong.
Multiple template variables in single property lead to an exception
I wanted to use multiple template variables in a single property:
{
// snip
"caller": "${json:source:className}.${json:source:methodName}",
"file": "${json:source:fileName}#${json:source:lineNumber}"
}The problem is that I'm presented with an exception upon startup:
2020-07-29 18:55:18,959 main ERROR Could not create plugin of type class com.vlkan.log4j2.logstash.layout.LogstashLayout for element LogstashLayout: java.lang.IllegalArgumentException: unknown key: className}.${json:source:methodName java.lang.IllegalArgumentException: unknown key: className}.${json:source:methodName
at com.vlkan.log4j2.logstash.layout.resolver.SourceResolver.createInternalResolver(SourceResolver.java:45)
at com.vlkan.log4j2.logstash.layout.resolver.SourceResolver.<init>(SourceResolver.java:32)
at com.vlkan.log4j2.logstash.layout.resolver.SourceResolverFactory.create(SourceResolverFactory.java:36)
at com.vlkan.log4j2.logstash.layout.resolver.SourceResolverFactory.create(SourceResolverFactory.java:19)
at com.vlkan.log4j2.logstash.layout.resolver.TemplateResolvers.ofStringNode(TemplateResolvers.java:168)
at com.vlkan.log4j2.logstash.layout.resolver.TemplateResolvers.ofNode(TemplateResolvers.java:80)
at com.vlkan.log4j2.logstash.layout.resolver.TemplateResolvers.ofObjectNode(TemplateResolvers.java:127)
at com.vlkan.log4j2.logstash.layout.resolver.TemplateResolvers.ofNode(TemplateResolvers.java:79)
at com.vlkan.log4j2.logstash.layout.resolver.TemplateResolvers.ofTemplate(TemplateResolvers.java:69)
at com.vlkan.log4j2.logstash.layout.LogstashLayout.createEventResolver(LogstashLayout.java:153)
at com.vlkan.log4j2.logstash.layout.LogstashLayout.<init>(LogstashLayout.java:85)
at com.vlkan.log4j2.logstash.layout.LogstashLayout.<init>(LogstashLayout.java:61)
at com.vlkan.log4j2.logstash.layout.LogstashLayout$Builder.build(LogstashLayout.java:547)
at com.vlkan.log4j2.logstash.layout.LogstashLayout$Builder.build(LogstashLayout.java:297)
at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:122)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:1002)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:942)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:934)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:934)
at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:552)
at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:241)
at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:288)
at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:618)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:691)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:708)
at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:263)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:153)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45)
at org.apache.logging.log4j.LogManager.getContext(LogManager.java:194)
at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:138)
at org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:45)
at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:48)
at org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:30)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:363)
at com.fleshgrinder.logging.LoggingKt.main(Logging.kt:6)
at com.fleshgrinder.logging.LoggingKt.main(Logging.kt)
2020-07-29 18:55:18,960 main ERROR Unable to invoke factory method in class com.vlkan.log4j2.logstash.layout.LogstashLayout for element LogstashLayout: java.lang.IllegalStateException: No factory method found for class com.vlkan.log4j2.logstash.layout.LogstashLayout java.lang.IllegalStateException: No factory method found for class com.vlkan.log4j2.logstash.layout.LogstashLayout
at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.findFactoryMethod(PluginBuilder.java:234)
at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:134)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:1002)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:942)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:934)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:934)
at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:552)
at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:241)
at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:288)
at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:618)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:691)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:708)
at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:263)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:153)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45)
at org.apache.logging.log4j.LogManager.getContext(LogManager.java:194)
at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:138)
at org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:45)
at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:48)
at org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:30)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:363)
at com.fleshgrinder.logging.LoggingKt.main(Logging.kt:6)
at com.fleshgrinder.logging.LoggingKt.main(Logging.kt)
test message
The moment I reduce the line to a single template variables things work as usual, however, I would expect this to work. ๐
PS: Thanks for the great library and your efforts to get it into core. โค๏ธ
Add entire map content as a field in the LogstashLayout
We're logging our messages via MapMessage through a LogstashLayout to ELK.
I'm trying to add the entire map content of the MapMessage to the layout but can't figure out how. I know that i can look up individual keys via
"event": "${map:event}",
but I haven't found a way to add the entire map.
I want to do something like:
{
"vars": "${map}"
}
and get sth. like
{
"vars": {
"key1": "value1",
"key2": "value2"
}
The message keys are more or less dynamic so i can't add them all as individual lookups.
From what I can see it's not possible to register a custom Resolver, isn't it?
Add a feature comparison table
Candidate layouts:
Add log4j2-ecs-layout to the benchmarks
@felixbarny has recently announced elastic/java-ecs-logging project which ships log4j2-ecs-layout artifact. Add this layout to the benchmarks.
Review request for Log4j JsonTemplateLayout
Dear past issue and pull request authors!
I am sorry to spam you in this way, though I need your help on something. I have been working extensively in the last one year to merge LogstashLayout into the Apache Log4j core. ๐ช The new layout is rebranded as JsonTemplateLayout (which also communicates the intent more clearly) and gone through a massive set of changes. ๐ I have uploaded the most recent version of the manual here. Your experience with the LogstashLayout makes you the perfect candidate for reviewing this new layout. I will really appreciate if you can spare some time to go through the manual and share your feedback. ๐ฅบ ๐ You can either write your comments here, e-mail me directly, or use the Log4j mailing lists.
Thanks so much for your support. Both LogstashLayout and JsonTemplateLayout wouldn't be possible without you all.
/cc @a1046149428 @Achaaab @bakomchik @billcchung @emschwar @ercpe @gjevardat @gopalkakularam @ilgrosso @J0nathan-S @justinsaliba @Marx2 @MikaelStrand @mkedwards @mzeijen @ov7a @pmackles @rgomezcasas @sabareeshkkanan @vtrbtf @waldeinburg @wangzhuo0978 @yskopets
Fallback for json:map: fields for libraries
Currently we're looking into adopting MapMessages for most of our logging needs, mapping the message fieldin the layout as, say json:map:message. This works fine for our code, but when dealing with library code that logs SimpleMessages we end up with empty message fields. Is there a workaround for this, some way to fallback a field from json:map: mapping to message if the mapping is empty?
NoClassDefFoundError: logstash.layout.resolver.TemplateResolvers (initialization failure)
Hi, I'm getting the following error when I use Logstash Layout.
The project is not a maven project but so I put the jar files manually.
Any idea why this happens?
java.lang.NoClassDefFoundError: com.vlkan.log4j2.logstash.layout.resolver.TemplateResolvers (initialization failure)
Full log:
[10/1/20 14:39:01:791 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,791 WebContainer : 1 INFO Log4j appears to be running in a Servlet environment, but there's no log4j-web module available. If you want better web container support, please add the log4j-web JAR to your web archive or server lib directory.
[10/1/20 14:39:01:795 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,795 WebContainer : 1 DEBUG Closing BufferedInputStream java.io.BufferedInputStream@40edbe51
[10/1/20 14:39:01:802 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,802 WebContainer : 1 DEBUG Apache Log4j Core 2.13.3 initializing configuration XmlConfiguration[location=/Was/IBM/WebSphere/AppServer/profiles/AppSrv01/installedApps/MYSERVER019Cell01/MyWebAppX.ear/MyWebAppX.war/WEB-INF/classes/log4j2.xml]
[10/1/20 14:39:01:803 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,803 WebContainer : 1 DEBUG Installed 1 script engine
[10/1/20 14:39:01:821 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,821 WebContainer : 1 DEBUG Oracle Nashorn version: 1.8.0, language: ECMAScript, threading: Not Thread Safe, compile: true, names: [nashorn, Nashorn, js, JS, JavaScript, javascript, ECMAScript, ecmascript], factory class: jdk.nashorn.api.scripting.NashornScriptEngineFactory
[10/1/20 14:39:01:822 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,822 WebContainer : 1 DEBUG PluginManager 'Core' found 125 plugins
[10/1/20 14:39:01:823 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,823 WebContainer : 1 DEBUG PluginManager 'Level' found 0 plugins
[10/1/20 14:39:01:826 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,825 WebContainer : 1 DEBUG Building Plugin[name=property, class=org.apache.logging.log4j.core.config.Property].
[10/1/20 14:39:01:827 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,826 WebContainer : 1 DEBUG createProperty(name="filead", value="/Was/sgk/logs/MyWebAppX/MyWebAppX")
[10/1/20 14:39:01:827 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,827 WebContainer : 1 DEBUG Building Plugin[name=property, class=org.apache.logging.log4j.core.config.Property].
[10/1/20 14:39:01:828 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,828 WebContainer : 1 DEBUG createProperty(name="env", value="test")
[10/1/20 14:39:01:828 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,828 WebContainer : 1 DEBUG Building Plugin[name=property, class=org.apache.logging.log4j.core.config.Property].
[10/1/20 14:39:01:829 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,829 WebContainer : 1 DEBUG createProperty(name="app", value="muhtasar")
[10/1/20 14:39:01:830 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,830 WebContainer : 1 DEBUG Building Plugin[name=property, class=org.apache.logging.log4j.core.config.Property].
[10/1/20 14:39:01:831 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,830 WebContainer : 1 DEBUG createProperty(name="module", value="gib")
[10/1/20 14:39:01:831 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,831 WebContainer : 1 DEBUG Building Plugin[name=property, class=org.apache.logging.log4j.core.config.Property].
[10/1/20 14:39:01:832 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,832 WebContainer : 1 DEBUG createProperty(name="appPassword", value="123")
[10/1/20 14:39:01:833 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,832 WebContainer : 1 DEBUG Building Plugin[name=property, class=org.apache.logging.log4j.core.config.Property].
[10/1/20 14:39:01:833 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,833 WebContainer : 1 DEBUG createProperty(name="kafkaVisioAdres", value="kafka1.intra:9092,kafka2.intra:9092,kafka3.intra:9092")
[10/1/20 14:39:01:834 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,834 WebContainer : 1 DEBUG Building Plugin[name=property, class=org.apache.logging.log4j.core.config.Property].
[10/1/20 14:39:01:835 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,834 WebContainer : 1 DEBUG createProperty(name="visologVersion", value="2")
[10/1/20 14:39:01:835 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,835 WebContainer : 1 DEBUG Building Plugin[name=property, class=org.apache.logging.log4j.core.config.Property].
[10/1/20 14:39:01:836 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,836 WebContainer : 1 DEBUG createProperty(name="locationInfoEnabled", value="true")
[10/1/20 14:39:01:836 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,836 WebContainer : 1 DEBUG Building Plugin[name=property, class=org.apache.logging.log4j.core.config.Property].
[10/1/20 14:39:01:837 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,837 WebContainer : 1 DEBUG createProperty(name="stackTraceEnabled", value="true")
[10/1/20 14:39:01:838 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,838 WebContainer : 1 DEBUG Building Plugin[name=property, class=org.apache.logging.log4j.core.config.Property].
[10/1/20 14:39:01:838 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,838 WebContainer : 1 DEBUG createProperty(name="syncSend", value="true")
[10/1/20 14:39:01:839 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,839 WebContainer : 1 DEBUG Building Plugin[name=property, class=org.apache.logging.log4j.core.config.Property].
[10/1/20 14:39:01:840 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,839 WebContainer : 1 DEBUG createProperty(name="acks", value="all")
[10/1/20 14:39:01:840 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,840 WebContainer : 1 DEBUG Building Plugin[name=property, class=org.apache.logging.log4j.core.config.Property].
[10/1/20 14:39:01:841 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,841 WebContainer : 1 DEBUG createProperty(name="retention1", value="2688")
[10/1/20 14:39:01:841 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,841 WebContainer : 1 DEBUG Building Plugin[name=property, class=org.apache.logging.log4j.core.config.Property].
[10/1/20 14:39:01:842 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,842 WebContainer : 1 DEBUG createProperty(name="timeZoneId", value="Asia/Istanbul")
[10/1/20 14:39:01:843 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,842 WebContainer : 1 DEBUG Building Plugin[name=properties, class=org.apache.logging.log4j.core.config.PropertiesPlugin].
[10/1/20 14:39:01:843 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,843 WebContainer : 1 DEBUG configureSubstitutor(={filead=/Was/sgk/logs/MyWebAppX/MyWebAppX, env=test, app=muhtasar, module=gib, appPassword=123, kafkaVisioAdres=kafka1.intra:9092,kafka2.intra:9092,kafka3.intra:9092, visologVersion=2, locationInfoEnabled=true, stackTraceEnabled=true, syncSend=true, acks=all, retention1=2688, timeZoneId=Asia/Istanbul}, Configuration(/Was/IBM/WebSphere/AppServer/profiles/AppSrv01/installedApps/MYSERVER019Cell01/MyWebAppX.ear/MyWebAppX.war/WEB-INF/classes/log4j2.xml))
[10/1/20 14:39:01:844 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,844 WebContainer : 1 DEBUG PluginManager 'Lookup' found 16 plugins
[10/1/20 14:39:01:847 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,847 WebContainer : 1 DEBUG Building Plugin[name=KeyValuePair, class=org.apache.logging.log4j.core.util.KeyValuePair].
[10/1/20 14:39:01:847 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,847 WebContainer : 1 DEBUG KeyValuePair$Builder(key="retention1", value="2688")
[10/1/20 14:39:01:848 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,848 WebContainer : 1 DEBUG Building Plugin[name=KeyValuePair, class=org.apache.logging.log4j.core.util.KeyValuePair].
[10/1/20 14:39:01:849 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,849 WebContainer : 1 DEBUG KeyValuePair$Builder(key="app", value="muhtasar")
[10/1/20 14:39:01:849 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,849 WebContainer : 1 DEBUG Building Plugin[name=KeyValuePair, class=org.apache.logging.log4j.core.util.KeyValuePair].
[10/1/20 14:39:01:850 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,850 WebContainer : 1 DEBUG KeyValuePair$Builder(key="module", value="gib")
[10/1/20 14:39:01:850 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,850 WebContainer : 1 DEBUG Building Plugin[name=KeyValuePair, class=org.apache.logging.log4j.core.util.KeyValuePair].
[10/1/20 14:39:01:851 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,850 WebContainer : 1 DEBUG KeyValuePair$Builder(key="visologVersion", value="2")
[10/1/20 14:39:01:851 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,851 WebContainer : 1 DEBUG Building Plugin[name=KeyValuePair, class=org.apache.logging.log4j.core.util.KeyValuePair].
[10/1/20 14:39:01:852 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,852 WebContainer : 1 DEBUG KeyValuePair$Builder(key="env", value="test")
[10/1/20 14:39:01:852 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,852 WebContainer : 1 DEBUG Building Plugin[name=EventTemplateAdditionalFields, class=com.vlkan.log4j2.logstash.layout.LogstashLayout$EventTemplateAdditionalFields].
[10/1/20 14:39:01:852 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,852 WebContainer : 1 DEBUG LogstashLayout$EventTemplateAdditionalFields$Builder(={retention1=2688, app=muhtasar, module=gib, visologVersion=2, env=test})
[10/1/20 14:39:01:853 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,853 WebContainer : 1 DEBUG Building Plugin[name=layout, class=com.vlkan.log4j2.logstash.layout.LogstashLayout].
[10/1/20 14:39:01:854 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,854 WebContainer : 1 DEBUG LogstashLayout$Builder(Configuration(/Was/IBM/WebSphere/AppServer/profiles/AppSrv01/installedApps/MYSERVER019Cell01/MyWebAppX.ear/MyWebAppX.war/WEB-INF/classes/log4j2.xml), prettyPrintEnabled="false", locationInfoEnabled="true", stackTraceEnabled="true", emptyPropertyExclusionEnabled="true", dateTimeFormatPattern="yyyy-MM-dd'T'HH:mm:ssZZZ", timeZoneId="Asia/Istanbul", locale="null", eventTemplate="null", eventTemplateUri="classpath:visologEventLayout.json", EventTemplateAdditionalFields(com.vlkan.log4j2.logstash.layout.LogstashLayout$EventTemplateAdditionalFields@8607a1f9), stackTraceElementTemplate="null", stackTraceElementTemplateUri="null", mdcKeyPattern="null", ndcPattern="null", lineSeparator="null", maxByteCount="32768", maxStringLength="null", objectMapperFactoryMethod="null", mapMessageFormatterIgnored="null")
[10/1/20 14:39:01:865 TRT] 000000c5 AxisEngine E org.apache.axis2.engine.AxisEngine receive null
org.apache.axis2.AxisFault
at org.apache.axis2.jaxws.server.dispatcher.JavaBeanDispatcher.createFaultResponse(JavaBeanDispatcher.java:407)
at org.apache.axis2.jaxws.server.dispatcher.JavaBeanDispatcher.invoke(JavaBeanDispatcher.java:138)
at org.apache.axis2.jaxws.server.EndpointController.invoke(EndpointController.java:111)
at org.apache.axis2.jaxws.server.JAXWSMessageReceiver.receive(JAXWSMessageReceiver.java:161)
at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:212)
at org.apache.axis2.transport.http.HTTPTransportUtils.processHTTPPostRequest(HTTPTransportUtils.java:172)
at com.ibm.ws.websvcs.transport.http.WASAxis2Servlet.doPost(WASAxis2Servlet.java:1633)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1235)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:779)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:478)
at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:179)
at com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1124)
at com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:82)
at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:963)
at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1817)
at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:382)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:465)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:532)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:318)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.ready(HttpInboundLink.java:289)
at com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.sendToDiscriminators(NewConnectionInitialReadCallback.java:214)
at com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.complete(NewConnectionInitialReadCallback.java:113)
at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:175)
at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)
at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)
at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)
at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775)
at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1909)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:90)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
at java.lang.reflect.Method.invoke(Method.java:508)
at org.apache.axis2.jaxws.server.dispatcher.JavaDispatcher.invokeTargetOperation(JavaDispatcher.java:120)
at org.apache.axis2.jaxws.server.dispatcher.JavaBeanDispatcher.invoke(JavaBeanDispatcher.java:118)
... 30 more
Caused by: java.lang.NoClassDefFoundError: com.vlkan.log4j2.logstash.layout.resolver.TemplateResolvers (initialization failure)
at java.lang.J9VMInternals.initializationAlreadyFailed(J9VMInternals.java:96)
at com.vlkan.log4j2.logstash.layout.LogstashLayout.createStackTraceElementResolver(LogstashLayout.java:121)
at com.vlkan.log4j2.logstash.layout.LogstashLayout.<init>(LogstashLayout.java:83)
at com.vlkan.log4j2.logstash.layout.LogstashLayout.<init>(LogstashLayout.java:61)
at com.vlkan.log4j2.logstash.layout.LogstashLayout$Builder.build(LogstashLayout.java:547)
at com.vlkan.log4j2.logstash.layout.LogstashLayout$Builder.build(LogstashLayout.java:297)
at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:122)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:1002)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:942)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:934)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:934)
at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:552)
at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:241)
at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:288)
at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:618)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:691)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:708)
at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:263)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:153)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45)
at org.apache.logging.log4j.LogManager.getContext(LogManager.java:194)
at org.apache.logging.log4j.LogManager.getLogger(LogManager.java:602)
at tr.gov.sgk.bildirge.webservice.business.PaketIslem.paketIptalSGK(PaketIslem.java:251)
at tr.gov.sgk.bildirge.webservice.BildirgeWS.paketIptal(BildirgeWS.java:316)
... 36 more
Caused by: java.lang.BootstrapMethodError: java.lang.LinkageError: loading constraint violation when overriding method "com/vlkan/log4j2/logstash/layout/resolver/TemplateResolver.resolve(Ljava/lang/Object;Lcom/fasterxml/jackson/core/JsonGenerator;)V" during creation of class "com/vlkan/log4j2/logstash/layout/resolver/TemplateResolvers$$Lambda$262/0000000008AB1080": loader "java/lang/InternalAnonymousClassLoader@e3c90ee7" of class "com/vlkan/log4j2/logstash/layout/resolver/TemplateResolvers$$Lambda$262/0000000008AB1080" and loader "com/ibm/ws/classloader/CompoundClassLoader@90ee7c7b" of class "com/vlkan/log4j2/logstash/layout/resolver/TemplateResolver" have different types for the method signature
at com.vlkan.log4j2.logstash.layout.resolver.TemplateResolvers.<clinit>(TemplateResolvers.java:34)
at com.vlkan.log4j2.logstash.layout.LogstashLayout.createStackTraceElementResolver(LogstashLayout.java:121)
at com.vlkan.log4j2.logstash.layout.LogstashLayout.<init>(LogstashLayout.java:83)
at com.vlkan.log4j2.logstash.layout.LogstashLayout.<init>(LogstashLayout.java:61)
at com.vlkan.log4j2.logstash.layout.LogstashLayout$Builder.build(LogstashLayout.java:547)
at com.vlkan.log4j2.logstash.layout.LogstashLayout$Builder.build(LogstashLayout.java:297)
at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:122)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:1002)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:942)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:934)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:934)
at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:552)
at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:241)
at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:288)
at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:618)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:691)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:708)
at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:263)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:243)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45)
at org.apache.logging.log4j.LogManager.getContext(LogManager.java:174)
at org.apache.logging.log4j.core.LoggerContext.getContext(LoggerContext.java:224)
at tr.gov.sgk.util.sistem.SistemSabitDegerler.log4j2Islemleri(SistemSabitDegerler.java:94)
at tr.gov.sgk.util.sistem.SistemSabitDegerler.<init>(SistemSabitDegerler.java:69)
at tr.gov.sgk.bildirge.webservice.util.StartServlet.init(StartServlet.java:44)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.init(ServletWrapper.java:342)
at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.init(ServletWrapperImpl.java:169)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.loadOnStartupCheck(ServletWrapper.java:1376)
at com.ibm.ws.webcontainer.webapp.WebApp.doLoadOnStartupActions(WebApp.java:673)
at com.ibm.ws.webcontainer.webapp.WebApp.commonInitializationFinally(WebApp.java:639)
at com.ibm.ws.webcontainer.webapp.WebAppImpl.initialize(WebAppImpl.java:454)
at com.ibm.ws.webcontainer.webapp.WebGroupImpl.addWebApplication(WebGroupImpl.java:88)
at com.ibm.ws.webcontainer.VirtualHostImpl.addWebApplication(VirtualHostImpl.java:171)
at com.ibm.ws.webcontainer.WSWebContainer.addWebApp(WSWebContainer.java:904)
at com.ibm.ws.webcontainer.WSWebContainer.addWebApplication(WSWebContainer.java:789)
at com.ibm.ws.webcontainer.component.WebContainerImpl.install(WebContainerImpl.java:427)
at com.ibm.ws.webcontainer.component.WebContainerImpl.start(WebContainerImpl.java:719)
at com.ibm.ws.runtime.component.ApplicationMgrImpl.start(ApplicationMgrImpl.java:1249)
at com.ibm.ws.runtime.component.DeployedApplicationImpl.fireDeployedObjectStart(DeployedApplicationImpl.java:1590)
at com.ibm.ws.runtime.component.DeployedModuleImpl.start(DeployedModuleImpl.java:707)
at com.ibm.ws.runtime.component.DeployedApplicationImpl.start(DeployedApplicationImpl.java:1161)
at com.ibm.ws.runtime.component.ApplicationMgrImpl.startApplication(ApplicationMgrImpl.java:801)
at com.ibm.ws.runtime.component.ApplicationMgrImpl.startApplicationDynamically(ApplicationMgrImpl.java:1451)
at com.ibm.ws.runtime.component.ApplicationMgrImpl.start(ApplicationMgrImpl.java:2319)
at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.start(CompositionUnitMgrImpl.java:436)
at com.ibm.ws.runtime.component.CompositionUnitImpl.start(CompositionUnitImpl.java:123)
at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.start(CompositionUnitMgrImpl.java:379)
at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.access$500(CompositionUnitMgrImpl.java:127)
at com.ibm.ws.runtime.component.CompositionUnitMgrImpl$1.run(CompositionUnitMgrImpl.java:654)
at com.ibm.ws.security.auth.ContextManagerImpl.runAs(ContextManagerImpl.java:5536)
at com.ibm.ws.security.auth.ContextManagerImpl.runAsSystem(ContextManagerImpl.java:5662)
at com.ibm.ws.security.core.SecurityContext.runAsSystem(SecurityContext.java:255)
at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.startCompositionUnit(CompositionUnitMgrImpl.java:668)
at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.startCompositionUnit(CompositionUnitMgrImpl.java:612)
at com.ibm.ws.runtime.component.ApplicationMgrImpl.startApplication(ApplicationMgrImpl.java:1341)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:90)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
at java.lang.reflect.Method.invoke(Method.java:508)
at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:83)
at sun.reflect.GeneratedMethodAccessor114.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
at java.lang.reflect.Method.invoke(Method.java:508)
at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:287)
at javax.management.modelmbean.RequiredModelMBean$4.run(RequiredModelMBean.java:1263)
at java.security.AccessController.doPrivileged(AccessController.java:664)
at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:91)
at javax.management.modelmbean.RequiredModelMBean.invokeMethod(RequiredModelMBean.java:1257)
at javax.management.modelmbean.RequiredModelMBean.invoke(RequiredModelMBean.java:1096)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:831)
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:813)
at com.ibm.ws.management.AdminServiceImpl$1.run(AdminServiceImpl.java:1350)
at com.ibm.ws.security.util.AccessController.doPrivileged(AccessController.java:118)
at com.ibm.ws.management.AdminServiceImpl.invoke(AdminServiceImpl.java:1243)
at com.ibm.ws.management.connector.AdminServiceDelegator.invoke(AdminServiceDelegator.java:181)
at com.ibm.ws.management.connector.ipc.CallRouter.route(CallRouter.java:247)
at com.ibm.ws.management.connector.ipc.IPCConnectorInboundLink.doWork(IPCConnectorInboundLink.java:360)
at com.ibm.ws.management.connector.ipc.IPCConnectorInboundLink$IPCConnectorReadCallback.complete(IPCConnectorInboundLink.java:602)
at com.ibm.ws.ssl.channel.impl.SSLReadServiceContext$SSLReadCompletedCallback.complete(SSLReadServiceContext.java:1833)
... 8 more
Caused by: java.lang.LinkageError: loading constraint violation when overriding method "com/vlkan/log4j2/logstash/layout/resolver/TemplateResolver.resolve(Ljava/lang/Object;Lcom/fasterxml/jackson/core/JsonGenerator;)V" during creation of class "com/vlkan/log4j2/logstash/layout/resolver/TemplateResolvers$$Lambda$262/0000000008AB1080": loader "java/lang/InternalAnonymousClassLoader@e3c90ee7" of class "com/vlkan/log4j2/logstash/layout/resolver/TemplateResolvers$$Lambda$262/0000000008AB1080" and loader "com/ibm/ws/classloader/CompoundClassLoader@90ee7c7b" of class "com/vlkan/log4j2/logstash/layout/resolver/TemplateResolver" have different types for the method signature
at sun.misc.Unsafe.defineAnonymousClass(Native Method)
at java.lang.invoke.InnerClassLambdaMetafactory.spinInnerClass(InnerClassLambdaMetafactory.java:339)
at java.lang.invoke.InnerClassLambdaMetafactory.buildCallSite(InnerClassLambdaMetafactory.java:206)
at java.lang.invoke.LambdaMetafactory.metafactory(LambdaMetafactory.java:315)
at java.lang.invoke.DirectHandle.invokeExact_thunkArchetype_L(DirectHandle.java:302)
at java.lang.invoke.AsTypeHandle.invokeExact_thunkArchetype_X(AsTypeHandle.java:49)
at java.lang.invoke.MethodHandle.resolveInvokeDynamic(MethodHandle.java:833)
... 87 more
[10/1/20 14:39:01:892 TRT] 000000c5 FfdcProvider W com.ibm.ws.ffdc.impl.FfdcProvider logIncident FFDC1003I: FFDC Incident emitted on /Was/IBM/WebSphere/AppServer/profiles/AppSrv01/logs/ffdc/uyg108_3_a5b084b7_20.10.01_14.39.01.8863351636182199464237.txt com.ibm.ws.websvcs.transport.http.WASAxis2Servlet.doPost 516
Ability to truncate individual fields
I think this is more of a feature request than a bug, but I am looking for a way to configure log4j2-logstash-layout so that it truncates certain fields if they are too long. It would be more granular than maxByteCount while also still allowing the message to go through. Something similar to the format modifiers in PatternLayout that allow you to truncate specific fields like message if it exceeds 10K characters. You can accomplish this through other means using logback but I can't find a way to do it in log4j2 short of a custom plugin.
If others agree with this, I can attempt a PR
BufferOverflowException get thrown for messages of a larger size
We have seen for many instances that the 1.0.0 version has a problem with messages of a larger size. It starts throwing BufferOverflowException exceptions when a certain threshold is reached. The exception that is logged to the console is as follows:
AsyncAppender-REDIS_GENERAL_LOG_ASYNC ERROR An exception occurred processing Appender REDIS_GENERAL_LOG java.lang.RuntimeException: failed serializing JSON
at com.vlkan.log4j2.logstash.layout.LogstashLayout.toByteArray(LogstashLayout.java:213)
at com.vlkan.log4j2.redis.appender.RedisAppender.append(RedisAppender.java:138)
at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:156)
at org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:129)
at org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:120)
at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:84)
at org.apache.logging.log4j.core.appender.AsyncAppender$AsyncThread.callAppenders(AsyncAppender.java:454)
at org.apache.logging.log4j.core.appender.AsyncAppender$AsyncThread.run(AsyncAppender.java:407)
Caused by: java.nio.BufferOverflowException
at java.base/java.nio.HeapByteBuffer.put(HeapByteBuffer.java:225)
at com.vlkan.log4j2.logstash.layout.util.ByteBufferOutputStream.write(ByteBufferOutputStream.java:48)
at com.vlkan.log4j2.logstash.layout.jackson.core.json.UTF8JsonGenerator._flushBuffer(UTF8JsonGenerator.java:2137)
at com.vlkan.log4j2.logstash.layout.jackson.core.json.UTF8JsonGenerator._writeStringSegment2(UTF8JsonGenerator.java:1413)
at com.vlkan.log4j2.logstash.layout.jackson.core.json.UTF8JsonGenerator._writeStringSegment(UTF8JsonGenerator.java:1366)
at com.vlkan.log4j2.logstash.layout.jackson.core.json.UTF8JsonGenerator._writeStringSegments(UTF8JsonGenerator.java:1307)
at com.vlkan.log4j2.logstash.layout.jackson.core.json.UTF8JsonGenerator.writeString(UTF8JsonGenerator.java:574)
at com.vlkan.log4j2.logstash.layout.resolver.StackTraceTextResolver.resolve(StackTraceTextResolver.java:43)
at com.vlkan.log4j2.logstash.layout.resolver.ExceptionRootCauseResolver$1.lambda$createStackTraceTextResolver$2(ExceptionRootCauseResolver.java:71)
at com.vlkan.log4j2.logstash.layout.resolver.ExceptionRootCauseResolver.resolve(ExceptionRootCauseResolver.java:103)
at com.vlkan.log4j2.logstash.layout.resolver.ExceptionRootCauseResolver.resolve(ExceptionRootCauseResolver.java:26)
at com.vlkan.log4j2.logstash.layout.resolver.TemplateResolvers.lambda$ofObjectNode$5(TemplateResolvers.java:146)
at com.vlkan.log4j2.logstash.layout.resolver.TemplateResolvers.lambda$ofObjectNode$5(TemplateResolvers.java:146)
at com.vlkan.log4j2.logstash.layout.resolver.TemplateResolvers.lambda$ofObjectNode$5(TemplateResolvers.java:146)
at com.vlkan.log4j2.logstash.layout.LogstashLayout.encode(LogstashLayout.java:262)
at com.vlkan.log4j2.logstash.layout.LogstashLayout.toByteArray(LogstashLayout.java:209)
I am not sure what the exact size is when this starts the happen. But testing with by increasing the message size has shown me that these exceptions starts the occur when the message contains +-15.700 characters. I believe the issue occurs when the total size of the serialised json data is getting to be, and that it is not specific to the message. We see these BufferOverflowException exceptions as well when a MDC field contains a large amount of data.
When reverting back to 0.21 we don't see this issue anymore
Improving the rendering performance
We think we're going to be able to open-source our message class fairly soon, at which point it might be interesting to explore alternate layout/message interfaces that reduce serialization overhead. Maybe something along the lines of a visitor pattern, where the Layout passes a visitor to the Message and the Message does the tree-walk? Some people refer to that as a SAX-style API, and Jackson's "Streaming API" (JsonGenerator, https://fasterxml.github.io/jackson-core/javadoc/2.9/com/fasterxml/jackson/core/JsonGenerator.html) works that way.
Have you worked at all with log4j2 upstream? It seems like this system would benefit from some hard-core perfomance work, which generally involves some API refactoring. If we could get a team together, it would be satisfying to build a "fish tagging" system for Java distributed architectures, comparable to what I've helped build previously in a Javascript/Python/C++ world. (That system resembled log4j2's ContextDataInjector scheme, with the local context transferred as metadata on RESTful API requests and auto-captured by task closures for delegation across thread boundaries.) Netty-based services really need such a thing.
Originally posted by @mkedwards in #15 (comment)
Stackoverflow "BOMB" after any unhandled exception
Hi guys!
recently i ran into problem , i got StackOverflowError and a lot of JsonGenerationException.
java.lang.StackOverflowError at com.fasterxml.jackson.core.filter.TokenFilterContext._writePath(TokenFilterContext.java:213) at com.fasterxml.jackson.core.filter.TokenFilterContext._writePath(TokenFilterContext.java:213) at com.fasterxml.jackson.core.filter.TokenFilterContext._writePath(TokenFilterContext.java:213) at com.fasterxml.jackson.core.filter.TokenFilterContext._writePath(TokenFilterContext.java:213) at com.fasterxml.jackson.core.filter.TokenFilterContext._writePath(TokenFilterContext.java:213) at com.fasterxml.jackson.core.filter.TokenFilterContext._writePath(TokenFilterContext.java:213) at com.fasterxml.jackson.core.filter.TokenFilterContext._writePath(TokenFilterContext.java:213) at com.fasterxml.jackson.core.filter.TokenFilterContext._writePath(TokenFilterContext.java:213) at com.fasterxml.jackson.core.filter.TokenFilterContext._writePath(TokenFilterContext.java:213) at com.fasterxml.jackson.core.filter.TokenFilterContext._writePath(TokenFilterContext.java:213)
After some investigation i found that after any non JsonGenerationException state of JsonGenerator become corrupted and after while leads to StackOverflowError.
I wrote test to demonstrate case, just follow the link bakomchik@f933ba0
I 've already made pull request.
#27
Use a faster timestamp serializer
ISO 8601 (e.g., 1970-01-01T00:00:00.000Z) timestamp formatting can be performed without any garbage. (See TimestampSerializer from java-ecs-logging.)
magic bug๏ผone line json become mutiple message
verison of 0.19
logstash 7.2
i use the tcp to logstash
<Socket name="logstash-tcp" host="${ctx:LOGSTASH_HOST}" port="${ctx:LOGSTASH_PORT}" protocol="TCP">
<LogstashLayout dateTimeFormatPattern="yyyy-MM-dd'T'HH:mm:ss.SSSZZZ"
eventTemplateUri="classpath:LogstashJsonEventLayoutV1.json"
prettyPrintEnabled="true"
stackTraceEnabled="true"/>
</Socket>
expected ,it shuoud be one line json
How exactly is MapMessage supposed to be handled?
Hi
I saw your bug report (and fix) for Log4J2's MapMessage handling. However, I've tried with a few different logging approaches and none seem to give me the output I was hoping for (where the MapMessage is output as a JSON Object).
Is there a little bit better instruction somewhere of how a MapMessage can be handled, or is it a limitation that the MapMessage class can't be serialized to an object?
Ultimately what I'm trying to do is log all headers into separate fields in my JSON log for an incoming request (to make each field searchable as well as run aggregates on different header values).
Take types into account while serializing MDC values
Hi man !
The lib is helping me a lot. However, it would be awesome if the layout could infer numeric values from MDC. This is something that would help on metrics and aggregations on ES.
Currently, we can only publish String MDC values.
If you're too busy, I could open a PR for that. However I dont know where to start
Vulnerable jackson version
The jackson-databind dependency of version 2.9.4 has a known critical vulnerability CVE-2018-7489.
Please upgrade to latest version 2.9.6 ๐
How can I change the output result type to a numeric type?
now, I can customize the output result through ThreadContext, config
"resTime": "${json:mdc:resTime}" , I get print
as {"resTime", "1000"}, but for some reason, I just want get {"resTime": 1000}, How could I do;
Use ByteArrayOutputStream instead of ByteBufferOutputStream to avoid message loss
I haven't researched this in depth but I suggest that the OutputStream in LogstashLayoutSerializationContexts is instantiated as a ByteArrayOutputStream (with an initial size of maxByteCount) instead of ByteBufferOutputStream to avoid message loss on large messages.
Setting maxStringLength is not a completely satisfying solution because we would have to make predictions of how many non-ASCII characters we expect as byte length is potentially larger than string length.
Failed serializing JSON: BufferOverflowException
Getting the exception below with GCP (using template from #54), version 1.0.2:
Log4j2-TF-1-AsyncLoggerConfig-1 ERROR An exception occurred processing Appender mainFile java.lang.RuntimeException: failed serializing JSON
at com.vlkan.log4j2.logstash.layout.LogstashLayout.encode(LogstashLayout.java:230)
at com.vlkan.log4j2.logstash.layout.LogstashLayout.encode(LogstashLayout.java:57)
at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.directEncodeEvent(AbstractOutputStreamAppender.java:197)
[...]
Caused by: java.nio.BufferOverflowException
at java.nio.HeapByteBuffer.put(HeapByteBuffer.java:194)
at com.vlkan.log4j2.logstash.layout.util.ByteBufferOutputStream.write(ByteBufferOutputStream.java:48)
at com.fasterxml.jackson.core.json.UTF8JsonGenerator._flushBuffer(UTF8JsonGenerator.java:2137)
at com.fasterxml.jackson.core.json.UTF8JsonGenerator.flush(UTF8JsonGenerator.java:1150)
at com.vlkan.log4j2.logstash.layout.LogstashLayout.encode(LogstashLayout.java:263)
at com.vlkan.log4j2.logstash.layout.LogstashLayout.encode(LogstashLayout.java:221)
... 19 more
MDC properties
org.apache.log4j.MDC properties not logged on the json file.
Is there a way to enable MDC on the LogstashLayout properties?
Close because using org.apache.logging.log4j.ThreadContext solves the problem.
sleuth logging pattern: pid and defaultvalues
Thank you very much for this outstanding plugin. As spring-sleuth is becomming more popular two minor issues with your logging-plugin are present.
first: provide pid
second: if one value is absent, then the plugin generates 'null' inside json whereas sleuth generates an empty string.
I hope you consider implementing these two minor enhancements
attacement: sleuth-pattern
Message lost if exceeding maxStringLength and emptyPropertyExclusionEnabled is true
If maxStringLength is exceeded and emptyPropertyExclusionEnabled is true, Jackson UTF8JsonGenerator will throw a JsonGenerationException: "Can not write a string, expecting field name (context: Object)".
Replace ByteBufferOutputStream with a dynamically growing one
When byteCount is big and not used at all (which is the case most of the time), ByteBufferOutputStream causes frequent cache misses. Use an alternative implementation with a dynamically growing byte[] alternative. (Maybe even chaining byte[]s via List<byte[]>?)
Messages as json for Logger.log(Level, Object) support
I am using Logger.log(Level, Object) and configured message as json:message:json in LogstashLayout. It only supports MultiformatMessage as json currently. Could you please provide support for any object as Json message as supported in JsonLayout of apache's log4j2 if we mention it as objectMessageAsJsonObject.
Also could you add support for the direct message should populated under messages instead of
message : { message: { JsonObject }}
Should be like: message: {JsonObject}
Improve LogstashJsonEventLayoutV1
Some ideas I wish log4j2-logstash-layout had:
- Rename
source_hostashostto be on par with Logstash Input File - Be able to use environment variables, Java system properties in a layout template:
- Use something like Log4J variable lookup to do property substitution https://logging.apache.org/log4j/2.0/manual/lookups.html
- Be able to add context specific fields like: "environment": "production" (yet avoiding the use of MDC which makes things complicated).
- Filter MDC content: include only a subset of MDC fields
- Make field names shorter "exception.class" instead of "exception.exception_class", "thread" instead of "thread_name"
This could be done with a different "enhanced" layout template
Replace Jackson with custom JSON emitter
While Jackson is the de facto JSON serialization library in the Java world and log4j2-logstash-layout enjoys its great performance, flexibility, and customization features, it is currently the slowest chain in the entire serialization pipeline. Given log4j2-logstash-layout only serializes LogEvents into JSON, a custom JSON emitter can very well be implemented in a pretty straightforward way.
Pros
- No external (required) dependency
- Faster serialization
Cons
- A
byte[]recycler will need to be implemented to avoid GC - Jackson is still needed as an optional dependency while serializing
Messages that are notSimpleMessage nullentry elimination might need to be dropped (difficult to get right)
@mkedwards Would you (or your company) be interested in sponsoring such an effort?
Support GELF layout
GELF layout can easily be supported by LogtashLayout. There are only two shortcomings:
- Rendering of epoch seconds with fraction. Such a feature can be introduced by means of a new
timestamp:divisor=<divisor>template, which will emitepochNanos / <divisor>as a floating point number. - Rendering of ordinal number of
Levels. This requires a newlevel:severityCodecode directive.
Once this is done, incorporating the LogstashLayout-powered GELF layout vs. Log4j 2 GelfLayout into benchmarks would be handy.
LogstashLayout with log4j2-elasticsearch appender
I'm trying to use logstash layout with project https://github.com/rfoltyns/log4j2-elasticsearch
<Elasticsearch name="elasticsearchAsyncBatch">
<IndexName indexName="log4j2"/>
<LogstashLayout dateTimeFormatPattern="yyyy-MM-dd'T'HH:mm:ss.SSSZZZ"
eventTemplateUri="classpath:EcsLayout.json"
prettyPrintEnabled="true"
stackTraceEnabled="true">
<EventTemplateAdditionalFields>
<KeyValuePair key="correlationId" value="$${CorrelationId:}"/>
<KeyValuePair key="serviceName" value="$${ServiceName:}"/>
<KeyValuePair key="environment" value="marx"/>
<KeyValuePair key="@timestamp" value="${date:yyyy-MM-dd HH:mm:ss.SSS}"/>
<KeyValuePair key="host_name" value="${hostName}"/>
</EventTemplateAdditionalFields>
</LogstashLayout>
<AsyncBatchDelivery>
<JestHttp serverUris="http://localhost:9200/_bulk"/>
</AsyncBatchDelivery>
</Elasticsearch>
However I'm getting error:
2020-01-13 15:25:20,025 main ERROR Could not create plugin of type class org.appenders.log4j2.elasticsearch.ElasticsearchAppender for element Elasticsearch: java.lang.IllegalArgumentException: Can not set org.apache.logging.log4j.core.layout.AbstractLayout field org.appenders.log4j2.elasticsearch.ElasticsearchAppender$Builder.layout to com.vlkan.log4j2.logstash.layout.LogstashLayout java.lang.IllegalArgumentException: Can not set org.apache.logging.log4j.core.layout.AbstractLayout field org.appenders.log4j2.elasticsearch.ElasticsearchAppender$Builder.layout to com.vlkan.log4j2.logstash.layout.LogstashLayout
at java.base/jdk.internal.reflect.UnsafeFieldAccessorImpl.throwSetIllegalArgumentException(UnsafeFieldAccessorImpl.java:167)
at java.base/jdk.internal.reflect.UnsafeFieldAccessorImpl.throwSetIllegalArgumentException(UnsafeFieldAccessorImpl.java:171)
at java.base/jdk.internal.reflect.UnsafeObjectFieldAccessorImpl.set(UnsafeObjectFieldAccessorImpl.java:81)
at java.base/java.lang.reflect.Field.set(Field.java:780)
at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.injectFields(PluginBuilder.java:188)
at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:121)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:964)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:904)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:896)
at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:514)
at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:238)
at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:250)
at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:548)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:620)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:637)
at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:231)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:153)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45)
at org.apache.logging.log4j.LogManager.getContext(LogManager.java:194)
at org.apache.commons.logging.LogAdapter$Log4jLog.<clinit>(LogAdapter.java:155)
at org.apache.commons.logging.LogAdapter$Log4jAdapter.createLog(LogAdapter.java:122)
at org.apache.commons.logging.LogAdapter.createLog(LogAdapter.java:89)
at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:67)
at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:59)
at org.springframework.boot.SpringApplication.<clinit>(SpringApplication.java:195)
at com.ydc.discovery.Application.main(Application.java:26)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.boot.maven.AbstractRunMojo$LaunchRunner.run(AbstractRunMojo.java:543)
at java.base/java.lang.Thread.run(Thread.java:831)
2020-01-13 15:25:20,036 main ERROR Unable to invoke factory method in class org.appenders.log4j2.elasticsearch.ElasticsearchAppender for element Elasticsearch: java.lang.IllegalStateException: No factory method found for class org.appenders.log4j2.elasticsearch.ElasticsearchAppender java.lang.IllegalStateException: No factory method found for class org.appenders.
Referenced issue: rfoltyns/log4j2-elasticsearch#32
LogStash Appender Question
I understand this project focuses only on the layout but I am wondering what appender to use instead of appending to a file and then uploading using logstash.
Remove max byte count cap?
I'm not sure if this is an issue or not, so just seeking clarification.
Is there a way to not cap the log size length? For our use case, we'd rather keep all the log messages instead of setting an arbitrary limit. Would it be an issue if we set the maxByte to something really high like 100k (our concern was whether these buffers were allocated for every log message)?
Introduce properties to globally set LogstashLayout.Builder defaults
For instance, -Dlog4j2.layout.logstash.maxByteCount.
Pretty printing in Logstash
I got parse issues with your extension, but stock log4j2 works with this config:
<JsonLayout compact="true" eventEol="true"/>
It inserts new line, I think you insert just a bunch of JSON nodes in the log file and he thinks it is an array, but there is no "," character.
My logstash config looks like this:
input {
file {
codec => json
type => "log4j-json"
path => "/logs/log4j2.log"
}
}
How to output ThreadContext (ex -MDC fields)
Hello
Thanks for you work, I'am using your plugin since one year or so, but without using ex-MDC fields because I could not understand quickly how to....
After reading the doc, it is not clear for me how to output some prefefined ThreadContext (ex MDC) fields. In my case these fields are simple strings or integers...
Is it fully in the log4j2 xml file or do I have to create a custom json template file?
If so how I instruct the appender to use this custom file?
Do you have an example please
Release Notes
Where to find release notes
Nullpointer exceptions at LogstashLayoutSerializationContexts$StringTruncatingGeneratorDelegate
We see Nullpointer exceptions with the following stack trace:
AsyncAppender-REDIS_GENERAL_LOG_ASYNC ERROR An exception occurred processing Appender REDIS_GENERAL_LOG java.lang.RuntimeException: failed serializing JSON
at com.vlkan.log4j2.logstash.layout.LogstashLayout.toByteArray(LogstashLayout.java:213)
at com.vlkan.log4j2.redis.appender.RedisAppender.append(RedisAppender.java:138)
at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:156)
at org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:129)
at org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:120)
at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:84)
at org.apache.logging.log4j.core.appender.AsyncAppender$AsyncThread.callAppenders(AsyncAppender.java:454)
at org.apache.logging.log4j.core.appender.AsyncAppender$AsyncThread.run(AsyncAppender.java:407)
Caused by: java.lang.NullPointerException
at com.vlkan.log4j2.logstash.layout.LogstashLayoutSerializationContexts$StringTruncatingGeneratorDelegate.writeString(LogstashLayoutSerializationContexts.java:128)
at com.vlkan.log4j2.logstash.layout.resolver.ExceptionResolver$1.lambda$createMessageResolver$1(ExceptionResolver.java:51)
at com.vlkan.log4j2.logstash.layout.resolver.ExceptionResolver.resolve(ExceptionResolver.java:98)
at com.vlkan.log4j2.logstash.layout.resolver.ExceptionResolver.resolve(ExceptionResolver.java:25)
at com.vlkan.log4j2.logstash.layout.resolver.TemplateResolvers.lambda$ofObjectNode$5(TemplateResolvers.java:146)
at com.vlkan.log4j2.logstash.layout.resolver.TemplateResolvers.lambda$ofObjectNode$5(TemplateResolvers.java:146)
at com.vlkan.log4j2.logstash.layout.resolver.TemplateResolvers.lambda$ofObjectNode$5(TemplateResolvers.java:146)
at com.vlkan.log4j2.logstash.layout.LogstashLayout.encode(LogstashLayout.java:262)
at com.vlkan.log4j2.logstash.layout.LogstashLayout.toByteArray(LogstashLayout.java:209)
... 7 more
If the StringTruncatingGeneratorDelegate#writeString method gets a null then it fails on the first if check, as it tries to call length on a null reference:
if (maxStringLength <= 0 || maxStringLength >= text.length()) {
super.writeString(text);
} else {
StringReader textReader = new StringReader(text);
super.writeString(textReader, maxStringLength);
}
It can get the null value from the ExceptionResolver when emptyPropertyExclusionEnabled is not set to true, which is the default.
I am not sure if this should be fixed in this write method or it needs to be fixed at the call sites of this method.
Stacktraces larger than maxStringLength are not truncated
Stacktraces larger than resolverContext.writerCapacity causes an ArrayIndexOutOfBoundsException in the call to System::arraycopy from BufferedWriter::write because it attempts to write past the end of the buffer. The message is lost.
For stacktraces, maxStringLength has different semantics than for normal values: Instead of causing truncation, the buffer capacity is set from maxStringLength instead of maxByteCount if set.
The minimal solution would be that BufferedWriter detects if it is going to write past the buffer size and truncates.
The ideal solution would be to truncate as described based on maxStringLength if set and also use a growable buffer to avoid message loss. You could use the same mechanism as ByteArrayOutputStream (c.f. the private method grow) instead of char[].
Add support for Custom Serializers
It would be nice, if there was support for custom serializers in order to be able to for example use org.json instead of fasterxml.
License
Hi there,
This project looks very useful but since it's licensed as GPL we can't use it. Would you be willing to relicense it as LGPL or some other license that does not require open sourcing?
Caused by: java.lang.RuntimeException: failed serializing JSON
Hi Volkan,
Since we use the JSONEventLayoutV0 of the Logstash implementation, I created my own template. Within the written log file, the JSON string is fragmented in many places and no longer valid. In our application logs I can see several exceptions. All exceptions refer to the used ByteBuffer.
log4j2.xml
<LogstashLayout dateTimeFormatPattern="yyyy-MM-dd'T'HH:mm:ss.SSS'Z'"
timeZoneId="UTC"
eventTemplateUri="classpath:JSONEventLayoutV0.json"
locationInfoEnabled="true"
prettyPrintEnabled="false"
stackTraceEnabled="true"/>JSONEventLayoutV0.json
{
"@version": null,
"@message": "${json:message}",
"@timestamp": "${json:timestamp}",
"@source_host": "${hostName}",
"@fields": {
"exception": {
"stacktrace": "${json:exception:stackTrace:text}",
"exception_class": "${json:exception:className}",
"exception_message": "${json:exception:message}"
},
"file": "${json:source:fileName}",
"method": "${json:source:methodName}",
"level": "${json:level}",
"line_number": "${json:source:lineNumber}",
"loggerName": "${json:logger:name}",
"class": "${json:source:className}",
"mdc": "${json:mdc}",
"ndc": "${json:ndc}",
"threadName": "${json:thread:name}"
}
}Stacktrace
org.apache.logging.log4j.core.appender.AppenderLoggingException: An exception occurred processing Appender RollingFileJson
at org.apache.logging.log4j.core.appender.DefaultErrorHandler.error(DefaultErrorHandler.java:75) ~[log4j-core-2.11.1.jar!/:2.11.1]
at org.apache.logging.log4j.core.config.AppenderControl.handleAppenderError(AppenderControl.java:165) ~[log4j-core-2.11.1.jar!/:2.11.1]
at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:158) ~[log4j-core-2.11.1.jar!/:2.11.1]
at org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:129) ~[log4j-core-2.11.1.jar!/:2.11.1]
at org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:120) ~[log4j-core-2.11.1.jar!/:2.11.1]
at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:84) ~[log4j-core-2.11.1.jar!/:2.11.1]
at org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:464) ~[log4j-core-2.11.1.jar!/:2.11.1]
at org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:448) ~[log4j-core-2.11.1.jar!/:2.11.1]
at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:431) ~[log4j-core-2.11.1.jar!/:2.11.1]
at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:406) ~[log4j-core-2.11.1.jar!/:2.11.1]
at org.apache.logging.log4j.core.config.AwaitCompletionReliabilityStrategy.log(AwaitCompletionReliabilityStrategy.java:63) ~[log4j-core-2.11.1.jar!/:2.11.1]
at org.apache.logging.log4j.core.Logger.logMessage(Logger.java:146) ~[log4j-core-2.11.1.jar!/:2.11.1]
at org.apache.logging.log4j.spi.AbstractLogger.tryLogMessage(AbstractLogger.java:2170) ~[log4j-api-2.11.1.jar!/:2.11.1]
at org.apache.logging.log4j.spi.AbstractLogger.logMessageTrackRecursion(AbstractLogger.java:2125) ~[log4j-api-2.11.1.jar!/:2.11.1]
at org.apache.logging.log4j.spi.AbstractLogger.logMessageSafely(AbstractLogger.java:2108) ~[log4j-api-2.11.1.jar!/:2.11.1]
at org.apache.logging.log4j.spi.AbstractLogger.logMessage(AbstractLogger.java:2007) ~[log4j-api-2.11.1.jar!/:2.11.1]
at org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:1866) ~[log4j-api-2.11.1.jar!/:2.11.1]
at org.apache.logging.slf4j.Log4jLogger.debug(Log4jLogger.java:119) ~[log4j-slf4j-impl-2.11.1.jar!/:2.11.1]
at com.mongodb.diagnostics.logging.SLF4JLogger.debug(SLF4JLogger.java:56) ~[mongo-java-driver-3.8.2.jar!/:?]
at com.mongodb.internal.connection.LoggingCommandEventSender.sendStartedEvent(LoggingCommandEventSender.java:68) ~[mongo-java-driver-3.8.2.jar!/:?]
at com.mongodb.internal.connection.InternalStreamConnection.sendAndReceive(InternalStreamConnection.java:246) ~[mongo-java-driver-3.8.2.jar!/:?]
at com.mongodb.internal.connection.CommandHelper.sendAndReceive(CommandHelper.java:83) ~[mongo-java-driver-3.8.2.jar!/:?]
at com.mongodb.internal.connection.CommandHelper.executeCommand(CommandHelper.java:38) ~[mongo-java-driver-3.8.2.jar!/:?]
at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitorRunnable.lookupServerDescription(DefaultServerMonitor.java:180) ~[mongo-java-driver-3.8.2.jar!/:?]
at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:124) ~[mongo-java-driver-3.8.2.jar!/:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
Caused by: java.lang.RuntimeException: failed serializing JSON
at com.vlkan.log4j2.logstash.layout.LogstashLayout.encode(LogstashLayout.java:143) ~[log4j2-logstash-layout-0.16.jar!/:?]
at com.vlkan.log4j2.logstash.layout.LogstashLayout.encode(LogstashLayout.java:34) ~[log4j2-logstash-layout-0.16.jar!/:?]
at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.directEncodeEvent(AbstractOutputStreamAppender.java:177) ~[log4j-core-2.11.1.jar!/:2.11.1]
at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.tryAppend(AbstractOutputStreamAppender.java:170) ~[log4j-core-2.11.1.jar!/:2.11.1]
at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.append(AbstractOutputStreamAppender.java:161) ~[log4j-core-2.11.1.jar!/:2.11.1]
at org.apache.logging.log4j.core.appender.RollingFileAppender.append(RollingFileAppender.java:309) ~[log4j-core-2.11.1.jar!/:2.11.1]
at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:156) ~[log4j-core-2.11.1.jar!/:2.11.1]
... 23 more
Caused by: java.lang.IllegalArgumentException: newPosition > limit: (547 > 8192)
at java.nio.Buffer.createPositionException(Buffer.java:318) ~[?:?]
at java.nio.Buffer.position(Buffer.java:293) ~[?:?]
at java.nio.ByteBuffer.position(ByteBuffer.java:1086) ~[?:?]
at java.nio.HeapByteBuffer.put(HeapByteBuffer.java:237) ~[?:?]
at org.apache.logging.log4j.core.layout.ByteBufferDestinationHelper.writeToUnsynchronized(ByteBufferDestinationHelper.java:47) ~[log4j-core-2.11.1.jar!/:2.11.1]
at com.vlkan.log4j2.logstash.layout.LogstashLayout.encode(LogstashLayout.java:141) ~[log4j2-logstash-layout-0.16.jar!/:?]
at com.vlkan.log4j2.logstash.layout.LogstashLayout.encode(LogstashLayout.java:34) ~[log4j2-logstash-layout-0.16.jar!/:?]
at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.directEncodeEvent(AbstractOutputStreamAppender.java:177) ~[log4j-core-2.11.1.jar!/:2.11.1]
at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.tryAppend(AbstractOutputStreamAppender.java:170) ~[log4j-core-2.11.1.jar!/:2.11.1]
at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.append(AbstractOutputStreamAppender.java:161) ~[log4j-core-2.11.1.jar!/:2.11.1]
at org.apache.logging.log4j.core.appender.RollingFileAppender.append(RollingFileAppender.java:309) ~[log4j-core-2.11.1.jar!/:2.11.1]
at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:156) ~[log4j-core-2.11.1.jar!/:2.11.1]
... 23 more
org.apache.logging.log4j.core.appender.AppenderLoggingException: An exception occurred processing Appender RollingFileJson
at org.apache.logging.log4j.core.appender.DefaultErrorHandler.error(DefaultErrorHandler.java:75) ~[log4j-core-2.11.1.jar!/:2.11.1]
at org.apache.logging.log4j.core.config.AppenderControl.handleAppenderError(AppenderControl.java:165) ~[log4j-core-2.11.1.jar!/:2.11.1]
at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:158) ~[log4j-core-2.11.1.jar!/:2.11.1]
at org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:129) ~[log4j-core-2.11.1.jar!/:2.11.1]
at org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:120) ~[log4j-core-2.11.1.jar!/:2.11.1]
at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:84) ~[log4j-core-2.11.1.jar!/:2.11.1]
at org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:464) ~[log4j-core-2.11.1.jar!/:2.11.1]
at org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:448) ~[log4j-core-2.11.1.jar!/:2.11.1]
at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:431) ~[log4j-core-2.11.1.jar!/:2.11.1]
at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:406) ~[log4j-core-2.11.1.jar!/:2.11.1]
at org.apache.logging.log4j.core.config.AwaitCompletionReliabilityStrategy.log(AwaitCompletionReliabilityStrategy.java:63) ~[log4j-core-2.11.1.jar!/:2.11.1]
at org.apache.logging.log4j.core.Logger.logMessage(Logger.java:146) ~[log4j-core-2.11.1.jar!/:2.11.1]
at org.apache.logging.slf4j.Log4jLogger.log(Log4jLogger.java:376) ~[log4j-slf4j-impl-2.11.1.jar!/:2.11.1]
at org.slf4j.bridge.SLF4JBridgeHandler.callLocationAwareLogger(SLF4JBridgeHandler.java:221) ~[jul-to-slf4j-1.7.25.jar!/:1.7.25]
at org.slf4j.bridge.SLF4JBridgeHandler.publish(SLF4JBridgeHandler.java:303) ~[jul-to-slf4j-1.7.25.jar!/:1.7.25]
at java.util.logging.Logger.log(Logger.java:979) ~[?:?]
at java.util.logging.Logger.doLog(Logger.java:1006) ~[?:?]
at java.util.logging.Logger.logp(Logger.java:1283) ~[?:?]
at org.apache.juli.logging.DirectJDKLog.log(DirectJDKLog.java:175) ~[tomcat-embed-core-9.0.12.jar!/:9.0.12]
at org.apache.juli.logging.DirectJDKLog.error(DirectJDKLog.java:141) ~[tomcat-embed-core-9.0.12.jar!/:9.0.12]
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:236) ~[tomcat-embed-core-9.0.12.jar!/:9.0.12]
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) ~[tomcat-embed-core-9.0.12.jar!/:9.0.12]
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:490) ~[tomcat-embed-core-9.0.12.jar!/:9.0.12]
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139) ~[tomcat-embed-core-9.0.12.jar!/:9.0.12]
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) ~[tomcat-embed-core-9.0.12.jar!/:9.0.12]
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) ~[tomcat-embed-core-9.0.12.jar!/:9.0.12]
at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:668) ~[tomcat-embed-core-9.0.12.jar!/:9.0.12]
at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:685) ~[tomcat-embed-core-9.0.12.jar!/:9.0.12]
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343) ~[tomcat-embed-core-9.0.12.jar!/:9.0.12]
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:408) ~[tomcat-embed-core-9.0.12.jar!/:9.0.12]
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) ~[tomcat-embed-core-9.0.12.jar!/:9.0.12]
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:770) ~[tomcat-embed-core-9.0.12.jar!/:9.0.12]
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1415) ~[tomcat-embed-core-9.0.12.jar!/:9.0.12]
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) ~[tomcat-embed-core-9.0.12.jar!/:9.0.12]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) ~[tomcat-embed-core-9.0.12.jar!/:9.0.12]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
Caused by: java.lang.RuntimeException: failed serializing JSON
at com.vlkan.log4j2.logstash.layout.LogstashLayout.encode(LogstashLayout.java:143) ~[log4j2-logstash-layout-0.16.jar!/:?]
at com.vlkan.log4j2.logstash.layout.LogstashLayout.encode(LogstashLayout.java:34) ~[log4j2-logstash-layout-0.16.jar!/:?]
at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.directEncodeEvent(AbstractOutputStreamAppender.java:177) ~[log4j-core-2.11.1.jar!/:2.11.1]
at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.tryAppend(AbstractOutputStreamAppender.java:170) ~[log4j-core-2.11.1.jar!/:2.11.1]
at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.append(AbstractOutputStreamAppender.java:161) ~[log4j-core-2.11.1.jar!/:2.11.1]
at org.apache.logging.log4j.core.appender.RollingFileAppender.append(RollingFileAppender.java:309) ~[log4j-core-2.11.1.jar!/:2.11.1]
at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:156) ~[log4j-core-2.11.1.jar!/:2.11.1]
... 35 more
Caused by: java.lang.IllegalArgumentException: newPosition > limit: (16384 > 8192)
at java.nio.Buffer.createPositionException(Buffer.java:318) ~[?:?]
at java.nio.Buffer.position(Buffer.java:293) ~[?:?]
at java.nio.ByteBuffer.position(ByteBuffer.java:1086) ~[?:?]
at java.nio.HeapByteBuffer.put(HeapByteBuffer.java:237) ~[?:?]
at org.apache.logging.log4j.core.layout.ByteBufferDestinationHelper.writeToUnsynchronized(ByteBufferDestinationHelper.java:43) ~[log4j-core-2.11.1.jar!/:2.11.1]
at com.vlkan.log4j2.logstash.layout.LogstashLayout.encode(LogstashLayout.java:141) ~[log4j2-logstash-layout-0.16.jar!/:?]
at com.vlkan.log4j2.logstash.layout.LogstashLayout.encode(LogstashLayout.java:34) ~[log4j2-logstash-layout-0.16.jar!/:?]
at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.directEncodeEvent(AbstractOutputStreamAppender.java:177) ~[log4j-core-2.11.1.jar!/:2.11.1]
at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.tryAppend(AbstractOutputStreamAppender.java:170) ~[log4j-core-2.11.1.jar!/:2.11.1]
at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.append(AbstractOutputStreamAppender.java:161) ~[log4j-core-2.11.1.jar!/:2.11.1]
at org.apache.logging.log4j.core.appender.RollingFileAppender.append(RollingFileAppender.java:309) ~[log4j-core-2.11.1.jar!/:2.11.1]
at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:156) ~[log4j-core-2.11.1.jar!/:2.11.1]
... 35 more
Backporting to an earlier version of Log4j2
Hi,
Thanks for taking the time to create this project. I'm currently looking at this library to add more verbose logging throughout my applications and this seems to be exactly what I'm looking for.
I've successfully demo'd this to my peers and it was very well received. The only snag which we encountered whilst testing this with our framework is that we are stuck on v0.14 of this library. This is because the library started using ByteBufferDestinationHelper in v0.15 and in later versions. This class was only introduced to Log4j2 in v2.9.
Would you be willing to consider downgrading this library to support Log4j2 2.8.2? I'm willing to do the work myself and submit a PR in the near future to support this, but before doing so I would like to understand whether you would be willing to provide sporadic feedback, or whether you would even consider this at all.
Thanks in advance,
Justin
add Marker support
The LogEvent template variables which can be used inside custom templates do not support Marker (LogEvent.getMarker()). Would be nice if this would be added.
Layout should be done as in the Log4j2-JSONLayout:
"marker" : {
"name" : "child",
"parents" : [ {
"name" : "parent",
"parents" : [ {
"name" : "grandparent"
} ]
} ]
},Environment variables, system properties in template
Be able to use environment variables, Java system properties in a layout template...
Maybe use something like Log4J variable lookup to do property substitution https://logging.apache.org/log4j/2.0/manual/lookups.html
No new lines after some time
Hi,
Using this configuration
<Console name="Console" target="SYSTEM_OUT"> <LogstashLayout dateTimeFormatPattern="yyyy-MM-dd'T'HH:mm:ss.SSS'Z'" templateUri="classpath:LogstashJsonEventLayoutV1.json" prettyPrintEnabled="false" stackTraceEnabled="true" timeZoneId="TimeZone.GMT_ID" /> </Console>
everything works as expected for the first several hundreds of lines: that is, one log message is printed per line (start and end with { and }).
After that it happens that one new line is omitted: so, two messages appear at a single line. After several more lines, new lines are not printed anymore and everything is appended to the last line of the log:
...ger"}{"@version":1,...
Any idea if it is a bug, or if I can work around it somehow?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. ๐๐๐
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google โค๏ธ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.