vy / log4j2-logstash-layout Goto Github PK
View Code? Open in Web Editor NEWLog4j 2.x plugin for customizable and Logstash-friendly JSON layout.
License: Apache License 2.0
Log4j 2.x plugin for customizable and Logstash-friendly JSON layout.
License: Apache License 2.0
i tried something like this but no luck
"env": "${spring.profiles.active}"
Triggered by a log4j-user discussion.
Well getting used to json log but it is really hard to read logs when there is error. Even though we are shipping it elastic search but it will be nice to have pretty print when the log level is error.
Ala Log4j 2 GelfLayout
, introduce messagePattern
configuration to support PatternLayout
while rendering messages.
I was wondering if there is any template available for usage with GCP, according to https://cloud.google.com/logging/docs/agent/configuration#process-payload
Thanks!
In the original logstash layout there is a possibility to use custom user fields:
https://github.com/logstash/log4j-jsonevent-layout#custom-user-fields
This feature is quite handy to provide, for example, a service name through log4j config.
I suppose one can do it with this lib, but It would require to use custom layout for each service which adds extra field with different value for each service. (Or maybe I didn't understand the docs). This way is not very convenient.
instead of having { "mdc" : { "customThreadContextField" : "someValue1", "secondCustomThreadContextField" : "someValue2", }, "@version" : 1, "source_host" : "raspberrypi", "message" : "some msg", }
I would like to have the mdc flat to the rest of the fields { "customThreadContextField" : "someValue1", "secondCustomThreadContextField" : "someValue2", "@version" : 1, "source_host" : "raspberrypi", "message" : "some msg", }
Is that somehow possible? I tried adding "customThreadContextField": "${mdc:customThreadContextField}",
to a custom layout template, but the syntax is obviously wrong.
I wanted to use multiple template variables in a single property:
{
// snip
"caller": "${json:source:className}.${json:source:methodName}",
"file": "${json:source:fileName}#${json:source:lineNumber}"
}
The problem is that I'm presented with an exception upon startup:
2020-07-29 18:55:18,959 main ERROR Could not create plugin of type class com.vlkan.log4j2.logstash.layout.LogstashLayout for element LogstashLayout: java.lang.IllegalArgumentException: unknown key: className}.${json:source:methodName java.lang.IllegalArgumentException: unknown key: className}.${json:source:methodName
at com.vlkan.log4j2.logstash.layout.resolver.SourceResolver.createInternalResolver(SourceResolver.java:45)
at com.vlkan.log4j2.logstash.layout.resolver.SourceResolver.<init>(SourceResolver.java:32)
at com.vlkan.log4j2.logstash.layout.resolver.SourceResolverFactory.create(SourceResolverFactory.java:36)
at com.vlkan.log4j2.logstash.layout.resolver.SourceResolverFactory.create(SourceResolverFactory.java:19)
at com.vlkan.log4j2.logstash.layout.resolver.TemplateResolvers.ofStringNode(TemplateResolvers.java:168)
at com.vlkan.log4j2.logstash.layout.resolver.TemplateResolvers.ofNode(TemplateResolvers.java:80)
at com.vlkan.log4j2.logstash.layout.resolver.TemplateResolvers.ofObjectNode(TemplateResolvers.java:127)
at com.vlkan.log4j2.logstash.layout.resolver.TemplateResolvers.ofNode(TemplateResolvers.java:79)
at com.vlkan.log4j2.logstash.layout.resolver.TemplateResolvers.ofTemplate(TemplateResolvers.java:69)
at com.vlkan.log4j2.logstash.layout.LogstashLayout.createEventResolver(LogstashLayout.java:153)
at com.vlkan.log4j2.logstash.layout.LogstashLayout.<init>(LogstashLayout.java:85)
at com.vlkan.log4j2.logstash.layout.LogstashLayout.<init>(LogstashLayout.java:61)
at com.vlkan.log4j2.logstash.layout.LogstashLayout$Builder.build(LogstashLayout.java:547)
at com.vlkan.log4j2.logstash.layout.LogstashLayout$Builder.build(LogstashLayout.java:297)
at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:122)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:1002)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:942)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:934)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:934)
at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:552)
at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:241)
at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:288)
at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:618)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:691)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:708)
at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:263)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:153)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45)
at org.apache.logging.log4j.LogManager.getContext(LogManager.java:194)
at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:138)
at org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:45)
at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:48)
at org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:30)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:363)
at com.fleshgrinder.logging.LoggingKt.main(Logging.kt:6)
at com.fleshgrinder.logging.LoggingKt.main(Logging.kt)
2020-07-29 18:55:18,960 main ERROR Unable to invoke factory method in class com.vlkan.log4j2.logstash.layout.LogstashLayout for element LogstashLayout: java.lang.IllegalStateException: No factory method found for class com.vlkan.log4j2.logstash.layout.LogstashLayout java.lang.IllegalStateException: No factory method found for class com.vlkan.log4j2.logstash.layout.LogstashLayout
at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.findFactoryMethod(PluginBuilder.java:234)
at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:134)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:1002)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:942)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:934)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:934)
at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:552)
at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:241)
at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:288)
at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:618)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:691)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:708)
at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:263)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:153)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45)
at org.apache.logging.log4j.LogManager.getContext(LogManager.java:194)
at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:138)
at org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:45)
at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:48)
at org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:30)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:363)
at com.fleshgrinder.logging.LoggingKt.main(Logging.kt:6)
at com.fleshgrinder.logging.LoggingKt.main(Logging.kt)
test message
The moment I reduce the line to a single template variables things work as usual, however, I would expect this to work. ๐
PS: Thanks for the great library and your efforts to get it into core. โค๏ธ
We're logging our messages via MapMessage
through a LogstashLayout
to ELK.
I'm trying to add the entire map content of the MapMessage
to the layout but can't figure out how. I know that i can look up individual keys via
"event": "${map:event}",
but I haven't found a way to add the entire map.
I want to do something like:
{
"vars": "${map}"
}
and get sth. like
{
"vars": {
"key1": "value1",
"key2": "value2"
}
The message keys are more or less dynamic so i can't add them all as individual lookups.
From what I can see it's not possible to register a custom Resolver
, isn't it?
Candidate layouts:
@felixbarny has recently announced elastic/java-ecs-logging project which ships log4j2-ecs-layout
artifact. Add this layout to the benchmarks.
Dear past issue and pull request authors!
I am sorry to spam you in this way, though I need your help on something. I have been working extensively in the last one year to merge LogstashLayout
into the Apache Log4j core. ๐ช The new layout is rebranded as JsonTemplateLayout
(which also communicates the intent more clearly) and gone through a massive set of changes. ๐ I have uploaded the most recent version of the manual here. Your experience with the LogstashLayout
makes you the perfect candidate for reviewing this new layout. I will really appreciate if you can spare some time to go through the manual and share your feedback. ๐ฅบ ๐ You can either write your comments here, e-mail me directly, or use the Log4j mailing lists.
Thanks so much for your support. Both LogstashLayout
and JsonTemplateLayout
wouldn't be possible without you all.
/cc @a1046149428 @Achaaab @bakomchik @billcchung @emschwar @ercpe @gjevardat @gopalkakularam @ilgrosso @J0nathan-S @justinsaliba @Marx2 @MikaelStrand @mkedwards @mzeijen @ov7a @pmackles @rgomezcasas @sabareeshkkanan @vtrbtf @waldeinburg @wangzhuo0978 @yskopets
Currently we're looking into adopting MapMessages for most of our logging needs, mapping the message
fieldin the layout as, say json:map:message
. This works fine for our code, but when dealing with library code that logs SimpleMessages we end up with empty message fields. Is there a workaround for this, some way to fallback a field from json:map:
mapping to message
if the mapping is empty?
Hi, I'm getting the following error when I use Logstash Layout.
The project is not a maven project but so I put the jar files manually.
Any idea why this happens?
java.lang.NoClassDefFoundError: com.vlkan.log4j2.logstash.layout.resolver.TemplateResolvers (initialization failure)
Full log:
[10/1/20 14:39:01:791 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,791 WebContainer : 1 INFO Log4j appears to be running in a Servlet environment, but there's no log4j-web module available. If you want better web container support, please add the log4j-web JAR to your web archive or server lib directory.
[10/1/20 14:39:01:795 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,795 WebContainer : 1 DEBUG Closing BufferedInputStream java.io.BufferedInputStream@40edbe51
[10/1/20 14:39:01:802 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,802 WebContainer : 1 DEBUG Apache Log4j Core 2.13.3 initializing configuration XmlConfiguration[location=/Was/IBM/WebSphere/AppServer/profiles/AppSrv01/installedApps/MYSERVER019Cell01/MyWebAppX.ear/MyWebAppX.war/WEB-INF/classes/log4j2.xml]
[10/1/20 14:39:01:803 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,803 WebContainer : 1 DEBUG Installed 1 script engine
[10/1/20 14:39:01:821 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,821 WebContainer : 1 DEBUG Oracle Nashorn version: 1.8.0, language: ECMAScript, threading: Not Thread Safe, compile: true, names: [nashorn, Nashorn, js, JS, JavaScript, javascript, ECMAScript, ecmascript], factory class: jdk.nashorn.api.scripting.NashornScriptEngineFactory
[10/1/20 14:39:01:822 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,822 WebContainer : 1 DEBUG PluginManager 'Core' found 125 plugins
[10/1/20 14:39:01:823 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,823 WebContainer : 1 DEBUG PluginManager 'Level' found 0 plugins
[10/1/20 14:39:01:826 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,825 WebContainer : 1 DEBUG Building Plugin[name=property, class=org.apache.logging.log4j.core.config.Property].
[10/1/20 14:39:01:827 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,826 WebContainer : 1 DEBUG createProperty(name="filead", value="/Was/sgk/logs/MyWebAppX/MyWebAppX")
[10/1/20 14:39:01:827 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,827 WebContainer : 1 DEBUG Building Plugin[name=property, class=org.apache.logging.log4j.core.config.Property].
[10/1/20 14:39:01:828 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,828 WebContainer : 1 DEBUG createProperty(name="env", value="test")
[10/1/20 14:39:01:828 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,828 WebContainer : 1 DEBUG Building Plugin[name=property, class=org.apache.logging.log4j.core.config.Property].
[10/1/20 14:39:01:829 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,829 WebContainer : 1 DEBUG createProperty(name="app", value="muhtasar")
[10/1/20 14:39:01:830 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,830 WebContainer : 1 DEBUG Building Plugin[name=property, class=org.apache.logging.log4j.core.config.Property].
[10/1/20 14:39:01:831 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,830 WebContainer : 1 DEBUG createProperty(name="module", value="gib")
[10/1/20 14:39:01:831 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,831 WebContainer : 1 DEBUG Building Plugin[name=property, class=org.apache.logging.log4j.core.config.Property].
[10/1/20 14:39:01:832 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,832 WebContainer : 1 DEBUG createProperty(name="appPassword", value="123")
[10/1/20 14:39:01:833 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,832 WebContainer : 1 DEBUG Building Plugin[name=property, class=org.apache.logging.log4j.core.config.Property].
[10/1/20 14:39:01:833 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,833 WebContainer : 1 DEBUG createProperty(name="kafkaVisioAdres", value="kafka1.intra:9092,kafka2.intra:9092,kafka3.intra:9092")
[10/1/20 14:39:01:834 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,834 WebContainer : 1 DEBUG Building Plugin[name=property, class=org.apache.logging.log4j.core.config.Property].
[10/1/20 14:39:01:835 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,834 WebContainer : 1 DEBUG createProperty(name="visologVersion", value="2")
[10/1/20 14:39:01:835 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,835 WebContainer : 1 DEBUG Building Plugin[name=property, class=org.apache.logging.log4j.core.config.Property].
[10/1/20 14:39:01:836 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,836 WebContainer : 1 DEBUG createProperty(name="locationInfoEnabled", value="true")
[10/1/20 14:39:01:836 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,836 WebContainer : 1 DEBUG Building Plugin[name=property, class=org.apache.logging.log4j.core.config.Property].
[10/1/20 14:39:01:837 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,837 WebContainer : 1 DEBUG createProperty(name="stackTraceEnabled", value="true")
[10/1/20 14:39:01:838 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,838 WebContainer : 1 DEBUG Building Plugin[name=property, class=org.apache.logging.log4j.core.config.Property].
[10/1/20 14:39:01:838 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,838 WebContainer : 1 DEBUG createProperty(name="syncSend", value="true")
[10/1/20 14:39:01:839 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,839 WebContainer : 1 DEBUG Building Plugin[name=property, class=org.apache.logging.log4j.core.config.Property].
[10/1/20 14:39:01:840 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,839 WebContainer : 1 DEBUG createProperty(name="acks", value="all")
[10/1/20 14:39:01:840 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,840 WebContainer : 1 DEBUG Building Plugin[name=property, class=org.apache.logging.log4j.core.config.Property].
[10/1/20 14:39:01:841 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,841 WebContainer : 1 DEBUG createProperty(name="retention1", value="2688")
[10/1/20 14:39:01:841 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,841 WebContainer : 1 DEBUG Building Plugin[name=property, class=org.apache.logging.log4j.core.config.Property].
[10/1/20 14:39:01:842 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,842 WebContainer : 1 DEBUG createProperty(name="timeZoneId", value="Asia/Istanbul")
[10/1/20 14:39:01:843 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,842 WebContainer : 1 DEBUG Building Plugin[name=properties, class=org.apache.logging.log4j.core.config.PropertiesPlugin].
[10/1/20 14:39:01:843 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,843 WebContainer : 1 DEBUG configureSubstitutor(={filead=/Was/sgk/logs/MyWebAppX/MyWebAppX, env=test, app=muhtasar, module=gib, appPassword=123, kafkaVisioAdres=kafka1.intra:9092,kafka2.intra:9092,kafka3.intra:9092, visologVersion=2, locationInfoEnabled=true, stackTraceEnabled=true, syncSend=true, acks=all, retention1=2688, timeZoneId=Asia/Istanbul}, Configuration(/Was/IBM/WebSphere/AppServer/profiles/AppSrv01/installedApps/MYSERVER019Cell01/MyWebAppX.ear/MyWebAppX.war/WEB-INF/classes/log4j2.xml))
[10/1/20 14:39:01:844 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,844 WebContainer : 1 DEBUG PluginManager 'Lookup' found 16 plugins
[10/1/20 14:39:01:847 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,847 WebContainer : 1 DEBUG Building Plugin[name=KeyValuePair, class=org.apache.logging.log4j.core.util.KeyValuePair].
[10/1/20 14:39:01:847 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,847 WebContainer : 1 DEBUG KeyValuePair$Builder(key="retention1", value="2688")
[10/1/20 14:39:01:848 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,848 WebContainer : 1 DEBUG Building Plugin[name=KeyValuePair, class=org.apache.logging.log4j.core.util.KeyValuePair].
[10/1/20 14:39:01:849 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,849 WebContainer : 1 DEBUG KeyValuePair$Builder(key="app", value="muhtasar")
[10/1/20 14:39:01:849 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,849 WebContainer : 1 DEBUG Building Plugin[name=KeyValuePair, class=org.apache.logging.log4j.core.util.KeyValuePair].
[10/1/20 14:39:01:850 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,850 WebContainer : 1 DEBUG KeyValuePair$Builder(key="module", value="gib")
[10/1/20 14:39:01:850 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,850 WebContainer : 1 DEBUG Building Plugin[name=KeyValuePair, class=org.apache.logging.log4j.core.util.KeyValuePair].
[10/1/20 14:39:01:851 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,850 WebContainer : 1 DEBUG KeyValuePair$Builder(key="visologVersion", value="2")
[10/1/20 14:39:01:851 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,851 WebContainer : 1 DEBUG Building Plugin[name=KeyValuePair, class=org.apache.logging.log4j.core.util.KeyValuePair].
[10/1/20 14:39:01:852 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,852 WebContainer : 1 DEBUG KeyValuePair$Builder(key="env", value="test")
[10/1/20 14:39:01:852 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,852 WebContainer : 1 DEBUG Building Plugin[name=EventTemplateAdditionalFields, class=com.vlkan.log4j2.logstash.layout.LogstashLayout$EventTemplateAdditionalFields].
[10/1/20 14:39:01:852 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,852 WebContainer : 1 DEBUG LogstashLayout$EventTemplateAdditionalFields$Builder(={retention1=2688, app=muhtasar, module=gib, visologVersion=2, env=test})
[10/1/20 14:39:01:853 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,853 WebContainer : 1 DEBUG Building Plugin[name=layout, class=com.vlkan.log4j2.logstash.layout.LogstashLayout].
[10/1/20 14:39:01:854 TRT] 000000c5 SystemOut O 2020-10-01 14:39:01,854 WebContainer : 1 DEBUG LogstashLayout$Builder(Configuration(/Was/IBM/WebSphere/AppServer/profiles/AppSrv01/installedApps/MYSERVER019Cell01/MyWebAppX.ear/MyWebAppX.war/WEB-INF/classes/log4j2.xml), prettyPrintEnabled="false", locationInfoEnabled="true", stackTraceEnabled="true", emptyPropertyExclusionEnabled="true", dateTimeFormatPattern="yyyy-MM-dd'T'HH:mm:ssZZZ", timeZoneId="Asia/Istanbul", locale="null", eventTemplate="null", eventTemplateUri="classpath:visologEventLayout.json", EventTemplateAdditionalFields(com.vlkan.log4j2.logstash.layout.LogstashLayout$EventTemplateAdditionalFields@8607a1f9), stackTraceElementTemplate="null", stackTraceElementTemplateUri="null", mdcKeyPattern="null", ndcPattern="null", lineSeparator="null", maxByteCount="32768", maxStringLength="null", objectMapperFactoryMethod="null", mapMessageFormatterIgnored="null")
[10/1/20 14:39:01:865 TRT] 000000c5 AxisEngine E org.apache.axis2.engine.AxisEngine receive null
org.apache.axis2.AxisFault
at org.apache.axis2.jaxws.server.dispatcher.JavaBeanDispatcher.createFaultResponse(JavaBeanDispatcher.java:407)
at org.apache.axis2.jaxws.server.dispatcher.JavaBeanDispatcher.invoke(JavaBeanDispatcher.java:138)
at org.apache.axis2.jaxws.server.EndpointController.invoke(EndpointController.java:111)
at org.apache.axis2.jaxws.server.JAXWSMessageReceiver.receive(JAXWSMessageReceiver.java:161)
at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:212)
at org.apache.axis2.transport.http.HTTPTransportUtils.processHTTPPostRequest(HTTPTransportUtils.java:172)
at com.ibm.ws.websvcs.transport.http.WASAxis2Servlet.doPost(WASAxis2Servlet.java:1633)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1235)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:779)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:478)
at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:179)
at com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1124)
at com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:82)
at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:963)
at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1817)
at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:382)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:465)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:532)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:318)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.ready(HttpInboundLink.java:289)
at com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.sendToDiscriminators(NewConnectionInitialReadCallback.java:214)
at com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.complete(NewConnectionInitialReadCallback.java:113)
at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:175)
at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)
at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)
at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)
at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775)
at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1909)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:90)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
at java.lang.reflect.Method.invoke(Method.java:508)
at org.apache.axis2.jaxws.server.dispatcher.JavaDispatcher.invokeTargetOperation(JavaDispatcher.java:120)
at org.apache.axis2.jaxws.server.dispatcher.JavaBeanDispatcher.invoke(JavaBeanDispatcher.java:118)
... 30 more
Caused by: java.lang.NoClassDefFoundError: com.vlkan.log4j2.logstash.layout.resolver.TemplateResolvers (initialization failure)
at java.lang.J9VMInternals.initializationAlreadyFailed(J9VMInternals.java:96)
at com.vlkan.log4j2.logstash.layout.LogstashLayout.createStackTraceElementResolver(LogstashLayout.java:121)
at com.vlkan.log4j2.logstash.layout.LogstashLayout.<init>(LogstashLayout.java:83)
at com.vlkan.log4j2.logstash.layout.LogstashLayout.<init>(LogstashLayout.java:61)
at com.vlkan.log4j2.logstash.layout.LogstashLayout$Builder.build(LogstashLayout.java:547)
at com.vlkan.log4j2.logstash.layout.LogstashLayout$Builder.build(LogstashLayout.java:297)
at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:122)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:1002)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:942)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:934)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:934)
at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:552)
at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:241)
at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:288)
at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:618)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:691)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:708)
at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:263)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:153)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45)
at org.apache.logging.log4j.LogManager.getContext(LogManager.java:194)
at org.apache.logging.log4j.LogManager.getLogger(LogManager.java:602)
at tr.gov.sgk.bildirge.webservice.business.PaketIslem.paketIptalSGK(PaketIslem.java:251)
at tr.gov.sgk.bildirge.webservice.BildirgeWS.paketIptal(BildirgeWS.java:316)
... 36 more
Caused by: java.lang.BootstrapMethodError: java.lang.LinkageError: loading constraint violation when overriding method "com/vlkan/log4j2/logstash/layout/resolver/TemplateResolver.resolve(Ljava/lang/Object;Lcom/fasterxml/jackson/core/JsonGenerator;)V" during creation of class "com/vlkan/log4j2/logstash/layout/resolver/TemplateResolvers$$Lambda$262/0000000008AB1080": loader "java/lang/InternalAnonymousClassLoader@e3c90ee7" of class "com/vlkan/log4j2/logstash/layout/resolver/TemplateResolvers$$Lambda$262/0000000008AB1080" and loader "com/ibm/ws/classloader/CompoundClassLoader@90ee7c7b" of class "com/vlkan/log4j2/logstash/layout/resolver/TemplateResolver" have different types for the method signature
at com.vlkan.log4j2.logstash.layout.resolver.TemplateResolvers.<clinit>(TemplateResolvers.java:34)
at com.vlkan.log4j2.logstash.layout.LogstashLayout.createStackTraceElementResolver(LogstashLayout.java:121)
at com.vlkan.log4j2.logstash.layout.LogstashLayout.<init>(LogstashLayout.java:83)
at com.vlkan.log4j2.logstash.layout.LogstashLayout.<init>(LogstashLayout.java:61)
at com.vlkan.log4j2.logstash.layout.LogstashLayout$Builder.build(LogstashLayout.java:547)
at com.vlkan.log4j2.logstash.layout.LogstashLayout$Builder.build(LogstashLayout.java:297)
at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:122)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:1002)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:942)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:934)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:934)
at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:552)
at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:241)
at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:288)
at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:618)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:691)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:708)
at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:263)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:243)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45)
at org.apache.logging.log4j.LogManager.getContext(LogManager.java:174)
at org.apache.logging.log4j.core.LoggerContext.getContext(LoggerContext.java:224)
at tr.gov.sgk.util.sistem.SistemSabitDegerler.log4j2Islemleri(SistemSabitDegerler.java:94)
at tr.gov.sgk.util.sistem.SistemSabitDegerler.<init>(SistemSabitDegerler.java:69)
at tr.gov.sgk.bildirge.webservice.util.StartServlet.init(StartServlet.java:44)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.init(ServletWrapper.java:342)
at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.init(ServletWrapperImpl.java:169)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.loadOnStartupCheck(ServletWrapper.java:1376)
at com.ibm.ws.webcontainer.webapp.WebApp.doLoadOnStartupActions(WebApp.java:673)
at com.ibm.ws.webcontainer.webapp.WebApp.commonInitializationFinally(WebApp.java:639)
at com.ibm.ws.webcontainer.webapp.WebAppImpl.initialize(WebAppImpl.java:454)
at com.ibm.ws.webcontainer.webapp.WebGroupImpl.addWebApplication(WebGroupImpl.java:88)
at com.ibm.ws.webcontainer.VirtualHostImpl.addWebApplication(VirtualHostImpl.java:171)
at com.ibm.ws.webcontainer.WSWebContainer.addWebApp(WSWebContainer.java:904)
at com.ibm.ws.webcontainer.WSWebContainer.addWebApplication(WSWebContainer.java:789)
at com.ibm.ws.webcontainer.component.WebContainerImpl.install(WebContainerImpl.java:427)
at com.ibm.ws.webcontainer.component.WebContainerImpl.start(WebContainerImpl.java:719)
at com.ibm.ws.runtime.component.ApplicationMgrImpl.start(ApplicationMgrImpl.java:1249)
at com.ibm.ws.runtime.component.DeployedApplicationImpl.fireDeployedObjectStart(DeployedApplicationImpl.java:1590)
at com.ibm.ws.runtime.component.DeployedModuleImpl.start(DeployedModuleImpl.java:707)
at com.ibm.ws.runtime.component.DeployedApplicationImpl.start(DeployedApplicationImpl.java:1161)
at com.ibm.ws.runtime.component.ApplicationMgrImpl.startApplication(ApplicationMgrImpl.java:801)
at com.ibm.ws.runtime.component.ApplicationMgrImpl.startApplicationDynamically(ApplicationMgrImpl.java:1451)
at com.ibm.ws.runtime.component.ApplicationMgrImpl.start(ApplicationMgrImpl.java:2319)
at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.start(CompositionUnitMgrImpl.java:436)
at com.ibm.ws.runtime.component.CompositionUnitImpl.start(CompositionUnitImpl.java:123)
at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.start(CompositionUnitMgrImpl.java:379)
at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.access$500(CompositionUnitMgrImpl.java:127)
at com.ibm.ws.runtime.component.CompositionUnitMgrImpl$1.run(CompositionUnitMgrImpl.java:654)
at com.ibm.ws.security.auth.ContextManagerImpl.runAs(ContextManagerImpl.java:5536)
at com.ibm.ws.security.auth.ContextManagerImpl.runAsSystem(ContextManagerImpl.java:5662)
at com.ibm.ws.security.core.SecurityContext.runAsSystem(SecurityContext.java:255)
at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.startCompositionUnit(CompositionUnitMgrImpl.java:668)
at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.startCompositionUnit(CompositionUnitMgrImpl.java:612)
at com.ibm.ws.runtime.component.ApplicationMgrImpl.startApplication(ApplicationMgrImpl.java:1341)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:90)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
at java.lang.reflect.Method.invoke(Method.java:508)
at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:83)
at sun.reflect.GeneratedMethodAccessor114.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
at java.lang.reflect.Method.invoke(Method.java:508)
at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:287)
at javax.management.modelmbean.RequiredModelMBean$4.run(RequiredModelMBean.java:1263)
at java.security.AccessController.doPrivileged(AccessController.java:664)
at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:91)
at javax.management.modelmbean.RequiredModelMBean.invokeMethod(RequiredModelMBean.java:1257)
at javax.management.modelmbean.RequiredModelMBean.invoke(RequiredModelMBean.java:1096)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:831)
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:813)
at com.ibm.ws.management.AdminServiceImpl$1.run(AdminServiceImpl.java:1350)
at com.ibm.ws.security.util.AccessController.doPrivileged(AccessController.java:118)
at com.ibm.ws.management.AdminServiceImpl.invoke(AdminServiceImpl.java:1243)
at com.ibm.ws.management.connector.AdminServiceDelegator.invoke(AdminServiceDelegator.java:181)
at com.ibm.ws.management.connector.ipc.CallRouter.route(CallRouter.java:247)
at com.ibm.ws.management.connector.ipc.IPCConnectorInboundLink.doWork(IPCConnectorInboundLink.java:360)
at com.ibm.ws.management.connector.ipc.IPCConnectorInboundLink$IPCConnectorReadCallback.complete(IPCConnectorInboundLink.java:602)
at com.ibm.ws.ssl.channel.impl.SSLReadServiceContext$SSLReadCompletedCallback.complete(SSLReadServiceContext.java:1833)
... 8 more
Caused by: java.lang.LinkageError: loading constraint violation when overriding method "com/vlkan/log4j2/logstash/layout/resolver/TemplateResolver.resolve(Ljava/lang/Object;Lcom/fasterxml/jackson/core/JsonGenerator;)V" during creation of class "com/vlkan/log4j2/logstash/layout/resolver/TemplateResolvers$$Lambda$262/0000000008AB1080": loader "java/lang/InternalAnonymousClassLoader@e3c90ee7" of class "com/vlkan/log4j2/logstash/layout/resolver/TemplateResolvers$$Lambda$262/0000000008AB1080" and loader "com/ibm/ws/classloader/CompoundClassLoader@90ee7c7b" of class "com/vlkan/log4j2/logstash/layout/resolver/TemplateResolver" have different types for the method signature
at sun.misc.Unsafe.defineAnonymousClass(Native Method)
at java.lang.invoke.InnerClassLambdaMetafactory.spinInnerClass(InnerClassLambdaMetafactory.java:339)
at java.lang.invoke.InnerClassLambdaMetafactory.buildCallSite(InnerClassLambdaMetafactory.java:206)
at java.lang.invoke.LambdaMetafactory.metafactory(LambdaMetafactory.java:315)
at java.lang.invoke.DirectHandle.invokeExact_thunkArchetype_L(DirectHandle.java:302)
at java.lang.invoke.AsTypeHandle.invokeExact_thunkArchetype_X(AsTypeHandle.java:49)
at java.lang.invoke.MethodHandle.resolveInvokeDynamic(MethodHandle.java:833)
... 87 more
[10/1/20 14:39:01:892 TRT] 000000c5 FfdcProvider W com.ibm.ws.ffdc.impl.FfdcProvider logIncident FFDC1003I: FFDC Incident emitted on /Was/IBM/WebSphere/AppServer/profiles/AppSrv01/logs/ffdc/uyg108_3_a5b084b7_20.10.01_14.39.01.8863351636182199464237.txt com.ibm.ws.websvcs.transport.http.WASAxis2Servlet.doPost 516
I think this is more of a feature request than a bug, but I am looking for a way to configure log4j2-logstash-layout so that it truncates certain fields if they are too long. It would be more granular than maxByteCount
while also still allowing the message to go through. Something similar to the format modifiers in PatternLayout that allow you to truncate specific fields like message
if it exceeds 10K characters. You can accomplish this through other means using logback but I can't find a way to do it in log4j2 short of a custom plugin.
If others agree with this, I can attempt a PR
We have seen for many instances that the 1.0.0
version has a problem with messages of a larger size. It starts throwing BufferOverflowException exceptions when a certain threshold is reached. The exception that is logged to the console is as follows:
AsyncAppender-REDIS_GENERAL_LOG_ASYNC ERROR An exception occurred processing Appender REDIS_GENERAL_LOG java.lang.RuntimeException: failed serializing JSON
at com.vlkan.log4j2.logstash.layout.LogstashLayout.toByteArray(LogstashLayout.java:213)
at com.vlkan.log4j2.redis.appender.RedisAppender.append(RedisAppender.java:138)
at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:156)
at org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:129)
at org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:120)
at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:84)
at org.apache.logging.log4j.core.appender.AsyncAppender$AsyncThread.callAppenders(AsyncAppender.java:454)
at org.apache.logging.log4j.core.appender.AsyncAppender$AsyncThread.run(AsyncAppender.java:407)
Caused by: java.nio.BufferOverflowException
at java.base/java.nio.HeapByteBuffer.put(HeapByteBuffer.java:225)
at com.vlkan.log4j2.logstash.layout.util.ByteBufferOutputStream.write(ByteBufferOutputStream.java:48)
at com.vlkan.log4j2.logstash.layout.jackson.core.json.UTF8JsonGenerator._flushBuffer(UTF8JsonGenerator.java:2137)
at com.vlkan.log4j2.logstash.layout.jackson.core.json.UTF8JsonGenerator._writeStringSegment2(UTF8JsonGenerator.java:1413)
at com.vlkan.log4j2.logstash.layout.jackson.core.json.UTF8JsonGenerator._writeStringSegment(UTF8JsonGenerator.java:1366)
at com.vlkan.log4j2.logstash.layout.jackson.core.json.UTF8JsonGenerator._writeStringSegments(UTF8JsonGenerator.java:1307)
at com.vlkan.log4j2.logstash.layout.jackson.core.json.UTF8JsonGenerator.writeString(UTF8JsonGenerator.java:574)
at com.vlkan.log4j2.logstash.layout.resolver.StackTraceTextResolver.resolve(StackTraceTextResolver.java:43)
at com.vlkan.log4j2.logstash.layout.resolver.ExceptionRootCauseResolver$1.lambda$createStackTraceTextResolver$2(ExceptionRootCauseResolver.java:71)
at com.vlkan.log4j2.logstash.layout.resolver.ExceptionRootCauseResolver.resolve(ExceptionRootCauseResolver.java:103)
at com.vlkan.log4j2.logstash.layout.resolver.ExceptionRootCauseResolver.resolve(ExceptionRootCauseResolver.java:26)
at com.vlkan.log4j2.logstash.layout.resolver.TemplateResolvers.lambda$ofObjectNode$5(TemplateResolvers.java:146)
at com.vlkan.log4j2.logstash.layout.resolver.TemplateResolvers.lambda$ofObjectNode$5(TemplateResolvers.java:146)
at com.vlkan.log4j2.logstash.layout.resolver.TemplateResolvers.lambda$ofObjectNode$5(TemplateResolvers.java:146)
at com.vlkan.log4j2.logstash.layout.LogstashLayout.encode(LogstashLayout.java:262)
at com.vlkan.log4j2.logstash.layout.LogstashLayout.toByteArray(LogstashLayout.java:209)
I am not sure what the exact size is when this starts the happen. But testing with by increasing the message size has shown me that these exceptions starts the occur when the message contains +-15.700 characters. I believe the issue occurs when the total size of the serialised json data is getting to be, and that it is not specific to the message. We see these BufferOverflowException exceptions as well when a MDC field contains a large amount of data.
When reverting back to 0.21 we don't see this issue anymore
We think we're going to be able to open-source our message class fairly soon, at which point it might be interesting to explore alternate layout/message interfaces that reduce serialization overhead. Maybe something along the lines of a visitor pattern, where the Layout passes a visitor to the Message and the Message does the tree-walk? Some people refer to that as a SAX-style API, and Jackson's "Streaming API" (JsonGenerator, https://fasterxml.github.io/jackson-core/javadoc/2.9/com/fasterxml/jackson/core/JsonGenerator.html) works that way.
Have you worked at all with log4j2 upstream? It seems like this system would benefit from some hard-core perfomance work, which generally involves some API refactoring. If we could get a team together, it would be satisfying to build a "fish tagging" system for Java distributed architectures, comparable to what I've helped build previously in a Javascript/Python/C++ world. (That system resembled log4j2's ContextDataInjector scheme, with the local context transferred as metadata on RESTful API requests and auto-captured by task closures for delegation across thread boundaries.) Netty-based services really need such a thing.
Originally posted by @mkedwards in #15 (comment)
Hi guys!
recently i ran into problem , i got StackOverflowError and a lot of JsonGenerationException.
java.lang.StackOverflowError at com.fasterxml.jackson.core.filter.TokenFilterContext._writePath(TokenFilterContext.java:213) at com.fasterxml.jackson.core.filter.TokenFilterContext._writePath(TokenFilterContext.java:213) at com.fasterxml.jackson.core.filter.TokenFilterContext._writePath(TokenFilterContext.java:213) at com.fasterxml.jackson.core.filter.TokenFilterContext._writePath(TokenFilterContext.java:213) at com.fasterxml.jackson.core.filter.TokenFilterContext._writePath(TokenFilterContext.java:213) at com.fasterxml.jackson.core.filter.TokenFilterContext._writePath(TokenFilterContext.java:213) at com.fasterxml.jackson.core.filter.TokenFilterContext._writePath(TokenFilterContext.java:213) at com.fasterxml.jackson.core.filter.TokenFilterContext._writePath(TokenFilterContext.java:213) at com.fasterxml.jackson.core.filter.TokenFilterContext._writePath(TokenFilterContext.java:213) at com.fasterxml.jackson.core.filter.TokenFilterContext._writePath(TokenFilterContext.java:213)
After some investigation i found that after any non JsonGenerationException state of JsonGenerator become corrupted and after while leads to StackOverflowError.
I wrote test to demonstrate case, just follow the link bakomchik@f933ba0
I 've already made pull request.
#27
ISO 8601 (e.g., 1970-01-01T00:00:00.000Z
) timestamp formatting can be performed without any garbage. (See TimestampSerializer from java-ecs-logging.)
verison of 0.19
logstash 7.2
i use the tcp to logstash
<Socket name="logstash-tcp" host="${ctx:LOGSTASH_HOST}" port="${ctx:LOGSTASH_PORT}" protocol="TCP">
<LogstashLayout dateTimeFormatPattern="yyyy-MM-dd'T'HH:mm:ss.SSSZZZ"
eventTemplateUri="classpath:LogstashJsonEventLayoutV1.json"
prettyPrintEnabled="true"
stackTraceEnabled="true"/>
</Socket>
expected ,it shuoud be one line json
Hi
I saw your bug report (and fix) for Log4J2's MapMessage handling. However, I've tried with a few different logging approaches and none seem to give me the output I was hoping for (where the MapMessage is output as a JSON Object).
Is there a little bit better instruction somewhere of how a MapMessage can be handled, or is it a limitation that the MapMessage class can't be serialized to an object?
Ultimately what I'm trying to do is log all headers into separate fields in my JSON log for an incoming request (to make each field searchable as well as run aggregates on different header values).
Hi man !
The lib is helping me a lot. However, it would be awesome if the layout could infer numeric values from MDC. This is something that would help on metrics and aggregations on ES.
Currently, we can only publish String MDC values.
If you're too busy, I could open a PR for that. However I dont know where to start
The jackson-databind
dependency of version 2.9.4 has a known critical vulnerability CVE-2018-7489.
Please upgrade to latest version 2.9.6 ๐
now, I can customize the output result through ThreadContext, config
"resTime": "${json:mdc:resTime}" , I get print
as {"resTime", "1000"}, but for some reason, I just want get {"resTime": 1000}, How could I do;
I haven't researched this in depth but I suggest that the OutputStream in LogstashLayoutSerializationContexts is instantiated as a ByteArrayOutputStream (with an initial size of maxByteCount) instead of ByteBufferOutputStream to avoid message loss on large messages.
Setting maxStringLength is not a completely satisfying solution because we would have to make predictions of how many non-ASCII characters we expect as byte length is potentially larger than string length.
Getting the exception below with GCP (using template from #54), version 1.0.2:
Log4j2-TF-1-AsyncLoggerConfig-1 ERROR An exception occurred processing Appender mainFile java.lang.RuntimeException: failed serializing JSON
at com.vlkan.log4j2.logstash.layout.LogstashLayout.encode(LogstashLayout.java:230)
at com.vlkan.log4j2.logstash.layout.LogstashLayout.encode(LogstashLayout.java:57)
at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.directEncodeEvent(AbstractOutputStreamAppender.java:197)
[...]
Caused by: java.nio.BufferOverflowException
at java.nio.HeapByteBuffer.put(HeapByteBuffer.java:194)
at com.vlkan.log4j2.logstash.layout.util.ByteBufferOutputStream.write(ByteBufferOutputStream.java:48)
at com.fasterxml.jackson.core.json.UTF8JsonGenerator._flushBuffer(UTF8JsonGenerator.java:2137)
at com.fasterxml.jackson.core.json.UTF8JsonGenerator.flush(UTF8JsonGenerator.java:1150)
at com.vlkan.log4j2.logstash.layout.LogstashLayout.encode(LogstashLayout.java:263)
at com.vlkan.log4j2.logstash.layout.LogstashLayout.encode(LogstashLayout.java:221)
... 19 more
org.apache.log4j.MDC
properties not logged on the json file.
Is there a way to enable MDC on the LogstashLayout
properties?
Close because using org.apache.logging.log4j.ThreadContext
solves the problem.
Thank you very much for this outstanding plugin. As spring-sleuth is becomming more popular two minor issues with your logging-plugin are present.
first: provide pid
second: if one value is absent, then the plugin generates 'null' inside json whereas sleuth generates an empty string.
I hope you consider implementing these two minor enhancements
attacement: sleuth-pattern
If maxStringLength is exceeded and emptyPropertyExclusionEnabled is true, Jackson UTF8JsonGenerator will throw a JsonGenerationException: "Can not write a string, expecting field name (context: Object)".
When byteCount
is big and not used at all (which is the case most of the time), ByteBufferOutputStream
causes frequent cache misses. Use an alternative implementation with a dynamically growing byte[]
alternative. (Maybe even chaining byte[]
s via List<byte[]>
?)
I am using Logger.log(Level, Object) and configured message as json:message:json in LogstashLayout. It only supports MultiformatMessage as json currently. Could you please provide support for any object as Json message as supported in JsonLayout of apache's log4j2 if we mention it as objectMessageAsJsonObject.
Also could you add support for the direct message should populated under messages instead of
message : { message: { JsonObject }}
Should be like: message: {JsonObject}
Some ideas I wish log4j2-logstash-layout had:
source_host
as host
to be on par with Logstash Input FileThis could be done with a different "enhanced" layout template
While Jackson is the de facto JSON serialization library in the Java world and log4j2-logstash-layout
enjoys its great performance, flexibility, and customization features, it is currently the slowest chain in the entire serialization pipeline. Given log4j2-logstash-layout
only serializes LogEvent
s into JSON, a custom JSON emitter can very well be implemented in a pretty straightforward way.
byte[]
recycler will need to be implemented to avoid GCMessage
s that are not SimpleMessage
null
entry elimination might need to be dropped (difficult to get right)@mkedwards Would you (or your company) be interested in sponsoring such an effort?
GELF layout can easily be supported by LogtashLayout
. There are only two shortcomings:
timestamp:divisor=<divisor>
template, which will emit epochNanos / <divisor>
as a floating point number.Level
s. This requires a new level:severityCode
code directive.Once this is done, incorporating the LogstashLayout
-powered GELF layout vs. Log4j 2 GelfLayout
into benchmarks would be handy.
I'm trying to use logstash layout with project https://github.com/rfoltyns/log4j2-elasticsearch
<Elasticsearch name="elasticsearchAsyncBatch">
<IndexName indexName="log4j2"/>
<LogstashLayout dateTimeFormatPattern="yyyy-MM-dd'T'HH:mm:ss.SSSZZZ"
eventTemplateUri="classpath:EcsLayout.json"
prettyPrintEnabled="true"
stackTraceEnabled="true">
<EventTemplateAdditionalFields>
<KeyValuePair key="correlationId" value="$${CorrelationId:}"/>
<KeyValuePair key="serviceName" value="$${ServiceName:}"/>
<KeyValuePair key="environment" value="marx"/>
<KeyValuePair key="@timestamp" value="${date:yyyy-MM-dd HH:mm:ss.SSS}"/>
<KeyValuePair key="host_name" value="${hostName}"/>
</EventTemplateAdditionalFields>
</LogstashLayout>
<AsyncBatchDelivery>
<JestHttp serverUris="http://localhost:9200/_bulk"/>
</AsyncBatchDelivery>
</Elasticsearch>
However I'm getting error:
2020-01-13 15:25:20,025 main ERROR Could not create plugin of type class org.appenders.log4j2.elasticsearch.ElasticsearchAppender for element Elasticsearch: java.lang.IllegalArgumentException: Can not set org.apache.logging.log4j.core.layout.AbstractLayout field org.appenders.log4j2.elasticsearch.ElasticsearchAppender$Builder.layout to com.vlkan.log4j2.logstash.layout.LogstashLayout java.lang.IllegalArgumentException: Can not set org.apache.logging.log4j.core.layout.AbstractLayout field org.appenders.log4j2.elasticsearch.ElasticsearchAppender$Builder.layout to com.vlkan.log4j2.logstash.layout.LogstashLayout
at java.base/jdk.internal.reflect.UnsafeFieldAccessorImpl.throwSetIllegalArgumentException(UnsafeFieldAccessorImpl.java:167)
at java.base/jdk.internal.reflect.UnsafeFieldAccessorImpl.throwSetIllegalArgumentException(UnsafeFieldAccessorImpl.java:171)
at java.base/jdk.internal.reflect.UnsafeObjectFieldAccessorImpl.set(UnsafeObjectFieldAccessorImpl.java:81)
at java.base/java.lang.reflect.Field.set(Field.java:780)
at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.injectFields(PluginBuilder.java:188)
at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:121)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:964)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:904)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:896)
at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:514)
at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:238)
at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:250)
at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:548)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:620)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:637)
at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:231)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:153)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45)
at org.apache.logging.log4j.LogManager.getContext(LogManager.java:194)
at org.apache.commons.logging.LogAdapter$Log4jLog.<clinit>(LogAdapter.java:155)
at org.apache.commons.logging.LogAdapter$Log4jAdapter.createLog(LogAdapter.java:122)
at org.apache.commons.logging.LogAdapter.createLog(LogAdapter.java:89)
at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:67)
at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:59)
at org.springframework.boot.SpringApplication.<clinit>(SpringApplication.java:195)
at com.ydc.discovery.Application.main(Application.java:26)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.boot.maven.AbstractRunMojo$LaunchRunner.run(AbstractRunMojo.java:543)
at java.base/java.lang.Thread.run(Thread.java:831)
2020-01-13 15:25:20,036 main ERROR Unable to invoke factory method in class org.appenders.log4j2.elasticsearch.ElasticsearchAppender for element Elasticsearch: java.lang.IllegalStateException: No factory method found for class org.appenders.log4j2.elasticsearch.ElasticsearchAppender java.lang.IllegalStateException: No factory method found for class org.appenders.
Referenced issue: rfoltyns/log4j2-elasticsearch#32
I understand this project focuses only on the layout but I am wondering what appender to use instead of appending to a file and then uploading using logstash.
I'm not sure if this is an issue or not, so just seeking clarification.
Is there a way to not cap the log size length? For our use case, we'd rather keep all the log messages instead of setting an arbitrary limit. Would it be an issue if we set the maxByte to something really high like 100k (our concern was whether these buffers were allocated for every log message)?
For instance, -Dlog4j2.layout.logstash.maxByteCount
.
I got parse issues with your extension, but stock log4j2 works with this config:
<JsonLayout compact="true" eventEol="true"/>
It inserts new line, I think you insert just a bunch of JSON nodes in the log file and he thinks it is an array, but there is no "," character.
My logstash config looks like this:
input {
file {
codec => json
type => "log4j-json"
path => "/logs/log4j2.log"
}
}
Hello
Thanks for you work, I'am using your plugin since one year or so, but without using ex-MDC fields because I could not understand quickly how to....
After reading the doc, it is not clear for me how to output some prefefined ThreadContext (ex MDC) fields. In my case these fields are simple strings or integers...
Is it fully in the log4j2 xml file or do I have to create a custom json template file?
If so how I instruct the appender to use this custom file?
Do you have an example please
Where to find release notes
We see Nullpointer exceptions with the following stack trace:
AsyncAppender-REDIS_GENERAL_LOG_ASYNC ERROR An exception occurred processing Appender REDIS_GENERAL_LOG java.lang.RuntimeException: failed serializing JSON
at com.vlkan.log4j2.logstash.layout.LogstashLayout.toByteArray(LogstashLayout.java:213)
at com.vlkan.log4j2.redis.appender.RedisAppender.append(RedisAppender.java:138)
at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:156)
at org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:129)
at org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:120)
at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:84)
at org.apache.logging.log4j.core.appender.AsyncAppender$AsyncThread.callAppenders(AsyncAppender.java:454)
at org.apache.logging.log4j.core.appender.AsyncAppender$AsyncThread.run(AsyncAppender.java:407)
Caused by: java.lang.NullPointerException
at com.vlkan.log4j2.logstash.layout.LogstashLayoutSerializationContexts$StringTruncatingGeneratorDelegate.writeString(LogstashLayoutSerializationContexts.java:128)
at com.vlkan.log4j2.logstash.layout.resolver.ExceptionResolver$1.lambda$createMessageResolver$1(ExceptionResolver.java:51)
at com.vlkan.log4j2.logstash.layout.resolver.ExceptionResolver.resolve(ExceptionResolver.java:98)
at com.vlkan.log4j2.logstash.layout.resolver.ExceptionResolver.resolve(ExceptionResolver.java:25)
at com.vlkan.log4j2.logstash.layout.resolver.TemplateResolvers.lambda$ofObjectNode$5(TemplateResolvers.java:146)
at com.vlkan.log4j2.logstash.layout.resolver.TemplateResolvers.lambda$ofObjectNode$5(TemplateResolvers.java:146)
at com.vlkan.log4j2.logstash.layout.resolver.TemplateResolvers.lambda$ofObjectNode$5(TemplateResolvers.java:146)
at com.vlkan.log4j2.logstash.layout.LogstashLayout.encode(LogstashLayout.java:262)
at com.vlkan.log4j2.logstash.layout.LogstashLayout.toByteArray(LogstashLayout.java:209)
... 7 more
If the StringTruncatingGeneratorDelegate#writeString
method gets a null then it fails on the first if check, as it tries to call length
on a null
reference:
if (maxStringLength <= 0 || maxStringLength >= text.length()) {
super.writeString(text);
} else {
StringReader textReader = new StringReader(text);
super.writeString(textReader, maxStringLength);
}
It can get the null
value from the ExceptionResolver
when emptyPropertyExclusionEnabled
is not set to true
, which is the default.
I am not sure if this should be fixed in this write method or it needs to be fixed at the call sites of this method.
Stacktraces larger than resolverContext.writerCapacity
causes an ArrayIndexOutOfBoundsException
in the call to System::arraycopy
from BufferedWriter::write
because it attempts to write past the end of the buffer. The message is lost.
For stacktraces, maxStringLength
has different semantics than for normal values: Instead of causing truncation, the buffer capacity is set from maxStringLength
instead of maxByteCount
if set.
The minimal solution would be that BufferedWriter
detects if it is going to write past the buffer size and truncates.
The ideal solution would be to truncate as described based on maxStringLength
if set and also use a growable buffer to avoid message loss. You could use the same mechanism as ByteArrayOutputStream
(c.f. the private method grow
) instead of char[]
.
It would be nice, if there was support for custom serializers in order to be able to for example use org.json instead of fasterxml.
Hi there,
This project looks very useful but since it's licensed as GPL we can't use it. Would you be willing to relicense it as LGPL or some other license that does not require open sourcing?
Hi Volkan,
Since we use the JSONEventLayoutV0 of the Logstash implementation, I created my own template. Within the written log file, the JSON string is fragmented in many places and no longer valid. In our application logs I can see several exceptions. All exceptions refer to the used ByteBuffer.
log4j2.xml
<LogstashLayout dateTimeFormatPattern="yyyy-MM-dd'T'HH:mm:ss.SSS'Z'"
timeZoneId="UTC"
eventTemplateUri="classpath:JSONEventLayoutV0.json"
locationInfoEnabled="true"
prettyPrintEnabled="false"
stackTraceEnabled="true"/>
JSONEventLayoutV0.json
{
"@version": null,
"@message": "${json:message}",
"@timestamp": "${json:timestamp}",
"@source_host": "${hostName}",
"@fields": {
"exception": {
"stacktrace": "${json:exception:stackTrace:text}",
"exception_class": "${json:exception:className}",
"exception_message": "${json:exception:message}"
},
"file": "${json:source:fileName}",
"method": "${json:source:methodName}",
"level": "${json:level}",
"line_number": "${json:source:lineNumber}",
"loggerName": "${json:logger:name}",
"class": "${json:source:className}",
"mdc": "${json:mdc}",
"ndc": "${json:ndc}",
"threadName": "${json:thread:name}"
}
}
Stacktrace
org.apache.logging.log4j.core.appender.AppenderLoggingException: An exception occurred processing Appender RollingFileJson
at org.apache.logging.log4j.core.appender.DefaultErrorHandler.error(DefaultErrorHandler.java:75) ~[log4j-core-2.11.1.jar!/:2.11.1]
at org.apache.logging.log4j.core.config.AppenderControl.handleAppenderError(AppenderControl.java:165) ~[log4j-core-2.11.1.jar!/:2.11.1]
at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:158) ~[log4j-core-2.11.1.jar!/:2.11.1]
at org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:129) ~[log4j-core-2.11.1.jar!/:2.11.1]
at org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:120) ~[log4j-core-2.11.1.jar!/:2.11.1]
at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:84) ~[log4j-core-2.11.1.jar!/:2.11.1]
at org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:464) ~[log4j-core-2.11.1.jar!/:2.11.1]
at org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:448) ~[log4j-core-2.11.1.jar!/:2.11.1]
at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:431) ~[log4j-core-2.11.1.jar!/:2.11.1]
at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:406) ~[log4j-core-2.11.1.jar!/:2.11.1]
at org.apache.logging.log4j.core.config.AwaitCompletionReliabilityStrategy.log(AwaitCompletionReliabilityStrategy.java:63) ~[log4j-core-2.11.1.jar!/:2.11.1]
at org.apache.logging.log4j.core.Logger.logMessage(Logger.java:146) ~[log4j-core-2.11.1.jar!/:2.11.1]
at org.apache.logging.log4j.spi.AbstractLogger.tryLogMessage(AbstractLogger.java:2170) ~[log4j-api-2.11.1.jar!/:2.11.1]
at org.apache.logging.log4j.spi.AbstractLogger.logMessageTrackRecursion(AbstractLogger.java:2125) ~[log4j-api-2.11.1.jar!/:2.11.1]
at org.apache.logging.log4j.spi.AbstractLogger.logMessageSafely(AbstractLogger.java:2108) ~[log4j-api-2.11.1.jar!/:2.11.1]
at org.apache.logging.log4j.spi.AbstractLogger.logMessage(AbstractLogger.java:2007) ~[log4j-api-2.11.1.jar!/:2.11.1]
at org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:1866) ~[log4j-api-2.11.1.jar!/:2.11.1]
at org.apache.logging.slf4j.Log4jLogger.debug(Log4jLogger.java:119) ~[log4j-slf4j-impl-2.11.1.jar!/:2.11.1]
at com.mongodb.diagnostics.logging.SLF4JLogger.debug(SLF4JLogger.java:56) ~[mongo-java-driver-3.8.2.jar!/:?]
at com.mongodb.internal.connection.LoggingCommandEventSender.sendStartedEvent(LoggingCommandEventSender.java:68) ~[mongo-java-driver-3.8.2.jar!/:?]
at com.mongodb.internal.connection.InternalStreamConnection.sendAndReceive(InternalStreamConnection.java:246) ~[mongo-java-driver-3.8.2.jar!/:?]
at com.mongodb.internal.connection.CommandHelper.sendAndReceive(CommandHelper.java:83) ~[mongo-java-driver-3.8.2.jar!/:?]
at com.mongodb.internal.connection.CommandHelper.executeCommand(CommandHelper.java:38) ~[mongo-java-driver-3.8.2.jar!/:?]
at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitorRunnable.lookupServerDescription(DefaultServerMonitor.java:180) ~[mongo-java-driver-3.8.2.jar!/:?]
at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:124) ~[mongo-java-driver-3.8.2.jar!/:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
Caused by: java.lang.RuntimeException: failed serializing JSON
at com.vlkan.log4j2.logstash.layout.LogstashLayout.encode(LogstashLayout.java:143) ~[log4j2-logstash-layout-0.16.jar!/:?]
at com.vlkan.log4j2.logstash.layout.LogstashLayout.encode(LogstashLayout.java:34) ~[log4j2-logstash-layout-0.16.jar!/:?]
at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.directEncodeEvent(AbstractOutputStreamAppender.java:177) ~[log4j-core-2.11.1.jar!/:2.11.1]
at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.tryAppend(AbstractOutputStreamAppender.java:170) ~[log4j-core-2.11.1.jar!/:2.11.1]
at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.append(AbstractOutputStreamAppender.java:161) ~[log4j-core-2.11.1.jar!/:2.11.1]
at org.apache.logging.log4j.core.appender.RollingFileAppender.append(RollingFileAppender.java:309) ~[log4j-core-2.11.1.jar!/:2.11.1]
at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:156) ~[log4j-core-2.11.1.jar!/:2.11.1]
... 23 more
Caused by: java.lang.IllegalArgumentException: newPosition > limit: (547 > 8192)
at java.nio.Buffer.createPositionException(Buffer.java:318) ~[?:?]
at java.nio.Buffer.position(Buffer.java:293) ~[?:?]
at java.nio.ByteBuffer.position(ByteBuffer.java:1086) ~[?:?]
at java.nio.HeapByteBuffer.put(HeapByteBuffer.java:237) ~[?:?]
at org.apache.logging.log4j.core.layout.ByteBufferDestinationHelper.writeToUnsynchronized(ByteBufferDestinationHelper.java:47) ~[log4j-core-2.11.1.jar!/:2.11.1]
at com.vlkan.log4j2.logstash.layout.LogstashLayout.encode(LogstashLayout.java:141) ~[log4j2-logstash-layout-0.16.jar!/:?]
at com.vlkan.log4j2.logstash.layout.LogstashLayout.encode(LogstashLayout.java:34) ~[log4j2-logstash-layout-0.16.jar!/:?]
at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.directEncodeEvent(AbstractOutputStreamAppender.java:177) ~[log4j-core-2.11.1.jar!/:2.11.1]
at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.tryAppend(AbstractOutputStreamAppender.java:170) ~[log4j-core-2.11.1.jar!/:2.11.1]
at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.append(AbstractOutputStreamAppender.java:161) ~[log4j-core-2.11.1.jar!/:2.11.1]
at org.apache.logging.log4j.core.appender.RollingFileAppender.append(RollingFileAppender.java:309) ~[log4j-core-2.11.1.jar!/:2.11.1]
at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:156) ~[log4j-core-2.11.1.jar!/:2.11.1]
... 23 more
org.apache.logging.log4j.core.appender.AppenderLoggingException: An exception occurred processing Appender RollingFileJson
at org.apache.logging.log4j.core.appender.DefaultErrorHandler.error(DefaultErrorHandler.java:75) ~[log4j-core-2.11.1.jar!/:2.11.1]
at org.apache.logging.log4j.core.config.AppenderControl.handleAppenderError(AppenderControl.java:165) ~[log4j-core-2.11.1.jar!/:2.11.1]
at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:158) ~[log4j-core-2.11.1.jar!/:2.11.1]
at org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:129) ~[log4j-core-2.11.1.jar!/:2.11.1]
at org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:120) ~[log4j-core-2.11.1.jar!/:2.11.1]
at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:84) ~[log4j-core-2.11.1.jar!/:2.11.1]
at org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:464) ~[log4j-core-2.11.1.jar!/:2.11.1]
at org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:448) ~[log4j-core-2.11.1.jar!/:2.11.1]
at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:431) ~[log4j-core-2.11.1.jar!/:2.11.1]
at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:406) ~[log4j-core-2.11.1.jar!/:2.11.1]
at org.apache.logging.log4j.core.config.AwaitCompletionReliabilityStrategy.log(AwaitCompletionReliabilityStrategy.java:63) ~[log4j-core-2.11.1.jar!/:2.11.1]
at org.apache.logging.log4j.core.Logger.logMessage(Logger.java:146) ~[log4j-core-2.11.1.jar!/:2.11.1]
at org.apache.logging.slf4j.Log4jLogger.log(Log4jLogger.java:376) ~[log4j-slf4j-impl-2.11.1.jar!/:2.11.1]
at org.slf4j.bridge.SLF4JBridgeHandler.callLocationAwareLogger(SLF4JBridgeHandler.java:221) ~[jul-to-slf4j-1.7.25.jar!/:1.7.25]
at org.slf4j.bridge.SLF4JBridgeHandler.publish(SLF4JBridgeHandler.java:303) ~[jul-to-slf4j-1.7.25.jar!/:1.7.25]
at java.util.logging.Logger.log(Logger.java:979) ~[?:?]
at java.util.logging.Logger.doLog(Logger.java:1006) ~[?:?]
at java.util.logging.Logger.logp(Logger.java:1283) ~[?:?]
at org.apache.juli.logging.DirectJDKLog.log(DirectJDKLog.java:175) ~[tomcat-embed-core-9.0.12.jar!/:9.0.12]
at org.apache.juli.logging.DirectJDKLog.error(DirectJDKLog.java:141) ~[tomcat-embed-core-9.0.12.jar!/:9.0.12]
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:236) ~[tomcat-embed-core-9.0.12.jar!/:9.0.12]
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) ~[tomcat-embed-core-9.0.12.jar!/:9.0.12]
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:490) ~[tomcat-embed-core-9.0.12.jar!/:9.0.12]
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139) ~[tomcat-embed-core-9.0.12.jar!/:9.0.12]
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) ~[tomcat-embed-core-9.0.12.jar!/:9.0.12]
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) ~[tomcat-embed-core-9.0.12.jar!/:9.0.12]
at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:668) ~[tomcat-embed-core-9.0.12.jar!/:9.0.12]
at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:685) ~[tomcat-embed-core-9.0.12.jar!/:9.0.12]
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343) ~[tomcat-embed-core-9.0.12.jar!/:9.0.12]
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:408) ~[tomcat-embed-core-9.0.12.jar!/:9.0.12]
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) ~[tomcat-embed-core-9.0.12.jar!/:9.0.12]
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:770) ~[tomcat-embed-core-9.0.12.jar!/:9.0.12]
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1415) ~[tomcat-embed-core-9.0.12.jar!/:9.0.12]
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) ~[tomcat-embed-core-9.0.12.jar!/:9.0.12]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) ~[tomcat-embed-core-9.0.12.jar!/:9.0.12]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
Caused by: java.lang.RuntimeException: failed serializing JSON
at com.vlkan.log4j2.logstash.layout.LogstashLayout.encode(LogstashLayout.java:143) ~[log4j2-logstash-layout-0.16.jar!/:?]
at com.vlkan.log4j2.logstash.layout.LogstashLayout.encode(LogstashLayout.java:34) ~[log4j2-logstash-layout-0.16.jar!/:?]
at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.directEncodeEvent(AbstractOutputStreamAppender.java:177) ~[log4j-core-2.11.1.jar!/:2.11.1]
at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.tryAppend(AbstractOutputStreamAppender.java:170) ~[log4j-core-2.11.1.jar!/:2.11.1]
at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.append(AbstractOutputStreamAppender.java:161) ~[log4j-core-2.11.1.jar!/:2.11.1]
at org.apache.logging.log4j.core.appender.RollingFileAppender.append(RollingFileAppender.java:309) ~[log4j-core-2.11.1.jar!/:2.11.1]
at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:156) ~[log4j-core-2.11.1.jar!/:2.11.1]
... 35 more
Caused by: java.lang.IllegalArgumentException: newPosition > limit: (16384 > 8192)
at java.nio.Buffer.createPositionException(Buffer.java:318) ~[?:?]
at java.nio.Buffer.position(Buffer.java:293) ~[?:?]
at java.nio.ByteBuffer.position(ByteBuffer.java:1086) ~[?:?]
at java.nio.HeapByteBuffer.put(HeapByteBuffer.java:237) ~[?:?]
at org.apache.logging.log4j.core.layout.ByteBufferDestinationHelper.writeToUnsynchronized(ByteBufferDestinationHelper.java:43) ~[log4j-core-2.11.1.jar!/:2.11.1]
at com.vlkan.log4j2.logstash.layout.LogstashLayout.encode(LogstashLayout.java:141) ~[log4j2-logstash-layout-0.16.jar!/:?]
at com.vlkan.log4j2.logstash.layout.LogstashLayout.encode(LogstashLayout.java:34) ~[log4j2-logstash-layout-0.16.jar!/:?]
at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.directEncodeEvent(AbstractOutputStreamAppender.java:177) ~[log4j-core-2.11.1.jar!/:2.11.1]
at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.tryAppend(AbstractOutputStreamAppender.java:170) ~[log4j-core-2.11.1.jar!/:2.11.1]
at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.append(AbstractOutputStreamAppender.java:161) ~[log4j-core-2.11.1.jar!/:2.11.1]
at org.apache.logging.log4j.core.appender.RollingFileAppender.append(RollingFileAppender.java:309) ~[log4j-core-2.11.1.jar!/:2.11.1]
at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:156) ~[log4j-core-2.11.1.jar!/:2.11.1]
... 35 more
Hi,
Thanks for taking the time to create this project. I'm currently looking at this library to add more verbose logging throughout my applications and this seems to be exactly what I'm looking for.
I've successfully demo'd this to my peers and it was very well received. The only snag which we encountered whilst testing this with our framework is that we are stuck on v0.14 of this library. This is because the library started using ByteBufferDestinationHelper
in v0.15 and in later versions. This class was only introduced to Log4j2 in v2.9.
Would you be willing to consider downgrading this library to support Log4j2 2.8.2? I'm willing to do the work myself and submit a PR in the near future to support this, but before doing so I would like to understand whether you would be willing to provide sporadic feedback, or whether you would even consider this at all.
Thanks in advance,
Justin
The LogEvent template variables which can be used inside custom templates do not support Marker (LogEvent.getMarker()). Would be nice if this would be added.
Layout should be done as in the Log4j2-JSONLayout:
"marker" : {
"name" : "child",
"parents" : [ {
"name" : "parent",
"parents" : [ {
"name" : "grandparent"
} ]
} ]
},
Be able to use environment variables, Java system properties in a layout template...
Maybe use something like Log4J variable lookup to do property substitution https://logging.apache.org/log4j/2.0/manual/lookups.html
Hi,
Using this configuration
<Console name="Console" target="SYSTEM_OUT"> <LogstashLayout dateTimeFormatPattern="yyyy-MM-dd'T'HH:mm:ss.SSS'Z'" templateUri="classpath:LogstashJsonEventLayoutV1.json" prettyPrintEnabled="false" stackTraceEnabled="true" timeZoneId="TimeZone.GMT_ID" /> </Console>
everything works as expected for the first several hundreds of lines: that is, one log message is printed per line (start and end with { and }).
After that it happens that one new line is omitted: so, two messages appear at a single line. After several more lines, new lines are not printed anymore and everything is appended to the last line of the log:
...ger"}{"@version":1,...
Any idea if it is a bug, or if I can work around it somehow?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.