log4j2-logstash-layout is not maintained anymore! Since Log4j 2.14.0, it is
superseded by log4j-layout-template-json shipping
JsonTemplateLayout,
which is a successor of LogstashLayout. We strongly advise all LogstashLayout
users to migrate to JsonTemplateLayout. We will do our best to bring bug fixes
to LogstashLayout, but all the new development efforts will be focused on
JsonTemplateLayout.
LogstashLayout is the fastest Log4j 2
JSON layout allowing schema customization and Logstash-friendly
output.
By default, LogstashLayout ships the official JSONEventLayoutV1 stated by
log4j-jsonevent-layout
Log4j 1.x plugin. Compared to
JSONLayout
included in Log4j 2 and log4j-jsonevent-layout, LogstashLayout provides
the following additional features:
- Superior performance
- Customizable JSON schema (see
eventTemplate[Uri]andstackTraceElementTemplate[Uri]parameters) - Customizable timestamp formatting (see
dateTimeFormatPatternandtimeZoneIdparameters)
Add the log4j2-logstash-layout dependency to your POM file
<dependency>
<groupId>com.vlkan.log4j2</groupId>
<artifactId>log4j2-logstash-layout</artifactId>
<version>${log4j2-logstash-layout.version}</version>
</dependency>in combination with a valid log4j-core dependency:
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-core</artifactId>
<version>${log4j2.version}</version>
</dependency>(Note that the Java 9 module name is com.vlkan.log4j2.logstash.layout.)
Below you can find a sample log4j2.xml snippet employing LogstashLayout.
<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="warn">
<Appenders>
<Console name="CONSOLE" target="SYSTEM_OUT">
<LogstashLayout dateTimeFormatPattern="yyyy-MM-dd'T'HH:mm:ss.SSSZZZ"
eventTemplateUri="classpath:LogstashJsonEventLayoutV1.json"
prettyPrintEnabled="true"
stackTraceEnabled="true"/>
</Console>
</Appenders>
<Loggers>
<Root level="info">
<AppenderRef ref="CONSOLE"/>
</Root>
</Loggers>
</Configuration>Or using the log4j2.properties file instead:
status = warn
appender.console.name = CONSOLE
appender.console.type = CONSOLE
appender.console.target = SYSTEM_OUT
appender.console.logstash.type = LogstashLayout
appender.console.logstash.dateTimeFormatPattern = yyyy-MM-dd'T'HH:mm:ss.SSSZZZ
appender.console.logstash.eventTemplateUri = classpath:LogstashJsonEventLayoutV1.json
appender.console.logstash.prettyPrintEnabled = true
appender.console.logstash.stackTraceEnabled = true
rootLogger.level = info
rootLogger.appenderRef.stdout.ref = CONSOLEThis generates an output as follows:
{
"exception": {
"exception_class": "java.lang.RuntimeException",
"exception_message": "test",
"stacktrace": "java.lang.RuntimeException: test\n\tat com.vlkan.log4j2.logstash.layout.demo.LogstashLayoutDemo.main(LogstashLayoutDemo.java:11)\n"
},
"line_number": 12,
"class": "com.vlkan.log4j2.logstash.layout.demo.LogstashLayoutDemo",
"@version": 1,
"source_host": "varlik",
"message": "Hello, error!",
"thread_name": "main",
"@timestamp": "2017-05-25T19:56:23.370+02:00",
"level": "ERROR",
"file": "LogstashLayoutDemo.java",
"method": "main",
"logger_name": "com.vlkan.log4j2.logstash.layout.demo.LogstashLayoutDemo"
}LogstashLayout is configured with the following parameters:
| Parameter Name | Type | Description |
|---|---|---|
prettyPrintEnabled |
boolean | enables pretty-printer (defaults to false) |
locationInfoEnabled |
boolean | includes the filename and line number in the output (defaults to false) |
stackTraceEnabled |
boolean | includes stack traces (defaults to false) |
emptyPropertyExclusionEnabled |
boolean | exclude empty and null properties (defaults to false) |
dateTimeFormatPattern |
String | timestamp formatter pattern (defaults to yyyy-MM-dd'T'HH:mm:ss.SSSZZZ) |
timeZoneId |
String | time zone id (defaults to TimeZone.getDefault().getID()) |
locale |
String | locale in one of the following forms: <language>, <language>_<country>, or <language>_<country>_<variant> (defaults to Locale.getDefault()) |
mdcKeyPattern |
String | regex to filter MDC keys (does not apply to direct mdc:key access) |
ndcPattern |
String | regex to filter NDC items |
eventTemplate |
String | inline JSON template for rendering LogEvents (has priority over eventTemplateUri) |
eventTemplateUri |
String | JSON template for rendering LogEvents (defaults to classpath:LogstashJsonEventLayoutV1.json) |
eventTemplateAdditionalFields1 |
KeyValuePair | additional key-value pairs appended to the root of the event template |
stackTraceElementTemplate |
String | inline JSON template for rendering StackTraceElements (has priority over stackTraceElementTemplateUri) |
stackTraceElementTemplateUri |
String | JSON template for rendering StackTraceElements (defaults to classpath:Log4j2StackTraceElementLayout.json) |
lineSeparator |
String | used to separate log outputs (defaults to System.lineSeparator()) |
maxByteCount |
int | used to cap the internal byte[] buffer used for serialization (defaults to 16 KiB) |
maxStringLength2 |
int | truncate string values longer than the specified limit (defaults to 0) |
objectMapperFactoryMethod |
String | custom object mapper factory method (defaults to com.fasterxml.jackson.databind.ObjectMapper.new) |
mapMessageFormatterIgnored |
boolean | as a temporary work around for LOG4J2-2703, serialize MapMessages using Jackson rather than MapMessage#getFormattedMessage() (defaults to true) |
1 One can configure additional event template fields as follows:
<LogstashLayout ...>
<EventTemplateAdditionalFields>
<KeyValuePair key="serviceName" value="auth-service"/>
<KeyValuePair key="containerId" value="6ede3f0ca7d9"/>
</EventTemplateAdditionalFields>
</LogstashLayout>2 Note that string value truncation via maxStringLength can take
place both in object keys and values, and this operation does not leave any
trace behind. maxStringLength is intended as a soft protection against bogus
input and one should always rely on maxByteCount for a hard limit.
eventTemplateUri denotes the URI pointing to the JSON template that will be used
while formatting the LogEvents. By default, LogstashLayout ships
LogstashJsonEventLayoutV1.json
providing the official Logstash JSONEventLayoutV1.
{
"mdc": "${json:mdc}",
"ndc": "${json:ndc}",
"exception": {
"exception_class": "${json:exception:className}",
"exception_message": "${json:exception:message}",
"stacktrace": "${json:exception:stackTrace:text}"
},
"line_number": "${json:source:lineNumber}",
"class": "${json:source:className}",
"@version": 1,
"source_host": "${hostName}",
"message": "${json:message}",
"thread_name": "${json:thread:name}",
"@timestamp": "${json:timestamp}",
"level": "${json:level}",
"file": "${json:source:fileName}",
"method": "${json:source:methodName}",
"logger_name": "${json:logger:name}"
}Similarly, stackTraceElementUri denotes the URI pointing to the JSON template
that will be used while formatting the StackTraceElements. By default,
LogstashLayout ships classpath:Log4j2StackTraceElementLayout.json
providing an identical stack trace structure produced by Log4j 2 JSONLayout.
{
"class": "${json:stackTraceElement:className}",
"method": "${json:stackTraceElement:methodName}",
"file": "${json:stackTraceElement:fileName}",
"line": "${json:stackTraceElement:lineNumber}"
}In case of need, you can create your own templates with a structure tailored
to your needs. That is, you can add new fields, remove or rename existing
ones, change the structure, etc. Please note that eventTemplateUri parameter
only supports file and classpath URI schemes.
Below is the list of allowed LogEvent template variables that will be replaced
while rendering the JSON output.
| Variable Name | Description |
|---|---|
endOfBatch |
logEvent.isEndOfBatch() |
exception:className |
logEvent.getThrown().getClass().getCanonicalName() |
exception:message |
logEvent.getThrown().getMessage() |
exception:stackTrace |
logEvent.getThrown().getStackTrace() (inactive when stackTraceEnabled=false) |
exception:stackTrace:text |
logEvent.getThrown().printStackTrace() (inactive when stackTraceEnabled=false) |
exceptionRootCause:className |
the innermost exception:className in causal chain |
exceptionRootCause:message |
the innermost exception:message in causal chain |
exceptionRootCause:stackTrace[:text] |
the innermost exception:stackTrace[:text] in causal chain |
level |
logEvent.getLevel() |
level:severity |
Syslog severity keyword of logEvent.getLevel() |
level:severity:code |
Syslog severity code of logEvent.getLevel() |
logger:fqcn |
logEvent.getLoggerFqcn() |
logger:name |
logEvent.getLoggerName() |
main:<key> |
performs Main Argument Lookup for the given key |
map:<key> |
performs Map Lookup for the given key |
marker:name |
logEvent.getMarker.getName() |
mdc |
Mapped Diagnostic Context Map<String, String> returned by logEvent.getContextData() |
mdc:<key> |
Mapped Diagnostic Context String associated with key (mdcKeyPattern is discarded) |
message |
logEvent.getFormattedMessage() |
message:json |
if logEvent.getMessage() is of type MultiformatMessage and supports JSON, its read value, otherwise, {"message": <formattedMessage>} object |
ndc |
Nested Diagnostic Context String[] returned by logEvent.getContextStack() |
source:className |
logEvent.getSource().getClassName() |
source:fileName |
logEvent.getSource().getFileName() (inactive when locationInfoEnabled=false) |
source:lineNumber |
logEvent.getSource().getLineNumber() (inactive when locationInfoEnabled=false) |
source:methodName |
logEvent.getSource().getMethodName() |
thread:id |
logEvent.getThreadId() |
thread:name |
logEvent.getThreadName() |
thread:priority |
logEvent.getThreadPriority() |
timestamp |
logEvent.getTimeMillis() formatted using dateTimeFormatPattern and timeZoneId |
timestamp:epoch |
epoch nanoseconds derived from logEvent.getInstant() |
timestamp:epoch:divisor=<divisor> |
epoch nanoseconds derived from logEvent.getInstant() divided by provided divisor (of type double) |
timestamp:epoch:divisor=<divisor>,integral |
epoch nanoseconds derived from logEvent.getInstant() divided by provided divisor (of type double) and casted to long |
JSON field lookups are performed using the ${json:<variable-name>} scheme
where <variable-name> is defined as <resolver-name>[:<resolver-key>].
Characters following colon (:) are treated as the resolver-key.
Log4j 2 Lookups
(e.g., ${java:version}, ${env:USER}, ${date:MM-dd-yyyy}) are supported
in templates too. Though note that while ${json:...} template variables are
expected to occupy an entire field, that is, "level": "${json:level}", a
lookup can be mixed within a regular text: "myCustomField": "Hello, ${env:USER}!".
Similarly, below is the list of allowed StackTraceElement template variables:
| Variable Name | Description |
|---|---|
stackTraceElement:className |
stackTraceElement.getClassName() |
stackTraceElement:methodName |
stackTraceElement.getMethodName() |
stackTraceElement:fileName |
stackTraceElement.getFileName() |
stackTraceElement:lineNumber |
stackTraceElement.getLineNumber() |
As in LogEvent templates, StackTraceElement templates support Log4j 2
lookups too.
See layout-demo directory for a sample application
demonstrating the usage of LogstashLayout.
log4j2-logstash-layout artifact contains the following predefined templates:
-
EcsLayout.jsondescribed by the Elastic Common Schema (ECS) specification -
LogstashJsonEventLayoutV1.jsondescribed in log4j-jsonevent-layout -
GelfLayout.jsondescribed by the Graylog Extended Log Format (GELF) payload specification with additional_threadand_loggerfields similar toGelfLayoutof Log4j 2 (Here it is advised to override the obligatoryhostfield with a user provided constant viaeventTemplateAdditionalFieldsto avoidhostNameproperty lookup at runtime, which incurs an extra cost.)
Below is a feature comparison matrix between LogstashLayout and alternatives.
| Feature | LogstashLayout |
JsonLayout |
EcsLayout |
|---|---|---|---|
| Java version | 8 | 73 | 6 |
| Dependencies | Jackson | Jackson | None |
| Full schema customization? | ✓ | ✕ | ✕ |
| Timestamp customization? | ✓ | ✕ | ✕ |
| (Almost) garbage-free? | ✓ | ✕ | ✓ |
Custom typed Message serialization? |
✓ | ✕ | ✓4 |
Custom typed MDC value serialization? |
✓ | ✕ | ✕ |
| Rendering stack traces as array? | ✓ | ✓ | ✓ |
| Enabling/Disabling JSON pretty print? | ✓ | ✓ | ✕ |
| Additional fields? | ✓ | ✓ | ✓ |
3 Log4j 2.4 and greater requires Java 7, versions 2.0-alpha1 to 2.3 required Java 6.
4 Only for ObjectMessages and if Jackson is in the classpath.
Project also contains a log4j2-logstash-layout-fatjar artifact which
includes all its transitive dependencies in a separate shaded package (to
avoid the JAR Hell) with the exception of log4j-core, that you need to
include separately.
This might come handy if you want to use this plugin along with already compiled applications, e.g., Elasticsearch 5.x and 6.x versions, which requires Log4j 2.
log4j2-logstash-layout is all about providing a highly customizable JSON
schema for your logs. Though this does not necessarily mean that all of its
features are expected to be supported by every appender in the market. For
instance, while prettyPrintEnabled=true works fine with
log4j2-redis-appender, it should be turned off
for Logstash's log4j-json file input type. (See
Pretty printing in Logstash issue.)
Make sure you configure log4j2-logstash-layout properly in a way that
is aligned with your appender of preference.
The source code contains a benchmark comparing LogstashLayout performance with
JsonLayout
(shipped by default in Log4j 2) and
EcsLayout
(shipped by Elastic). There JMH
is used to assess the rendering performance of these layouts. In the tests,
different LogEvent profiles are employed:
- full:
LogEventcontains MDC, NDC, and an exception. - lite:
LogEventhas no MDC, NDC, or exception attachment.
To give an idea, we ran the benchmark with the following settings:
- CPU: Intel i7 2.70GHz (x86-64, confined
javaprocess to a single core usingtaskset -c 0) - JVM: OpenJDK 64-Bit, AdoptOpenJDK, build 25.232-b09
-XX:+TieredCompilation-Dlog4j2.garbagefreeThreadContextMap=true-Dlog4j2.enableDirectEncoders=true-Dlog4j2.enable.threadlocals=true-Dlog4j2.is.webapp=false
- OS: Xubuntu 18.04.3 (4.15.0-70-generic, x86-64)
LogstashLayout4{Ecs,Json,Gelf}Layoutused default settings with the following exceptions:stackTraceEnabled:truemaxByteCount: (4096) 4KiB
JsonLayoutused in two different flavors:DefaultJsonLayout: default settingsCustomJsonLayout: default settings with an additional"@version": 1field (this forces instantiation of a wrapper class to obtain the necessary Jackson view)
EcsLayoutused with the following configurations:serviceName:benchmarkadditionalFields:new KeyValuePair[0]
GelfLayoutused with the following configurations:compressionType:off
The figures for serializing 1,000 LogEvents at each operation are as follows.
(See layout-benchmark directory for the full report.)
| Benchmark | ops/sec5 | B/op5 | |
|---|---|---|---|
| liteLogstashLayout4GelfLayout | 1,517,062 | ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ (100%) | 0.0 |
| liteLogstashLayout4EcsLayout | 1,196,255 | ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ (79%) | 0.0 |
| liteGelfLayout | 1,184,922 | ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ (78%) | 0.0 |
| liteLogstashLayout4JsonLayout | 870,012 | ▉▉▉▉▉▉▉▉▉▉▉ (57%) | 0.0 |
| liteEcsLayout | 836,648 | ▉▉▉▉▉▉▉▉▉▉▉ (55%) | 0.0 |
| liteDefaultJsonLayout | 506,985 | ▉▉▉▉▉▉▉ (33%) | 5,331,680.0 |
| liteCustomJsonLayout | 446,243 | ▉▉▉▉▉▉ (29%) | 5,740,400.0 |
| fullLogstashLayout4JsonLayout | 118,294 | ▉▉ (8%) | 104,000.1 |
| fullLogstashLayout4GelfLayout | 73,102 | ▉ (5%) | 35,663,200.3 |
| fullLogstashLayout4EcsLayout | 60,569 | ▉ (4%) | 35,631,200.4 |
| fullEcsLayout | 27,887 | ▉ (2%) | 46,479,200.5 |
| fullGelfLayout | 21,458 | ▉ (1%) | 58,911,200.7 |
| fullDefaultJsonLayout | 13,513 | ▉ (1%) | 234,102,401.5 |
| fullCustomJsonLayout | 13,511 | ▉ (1%) | 234,238,401.5 |
5 99th percentile
Let us try to answer some common questions:
-
How come
log4j2-logstash-layoutcan yield superior performance compared to Log4j 2JsonLayout? Log4j 2JsonLayoutemploys a single Jackson view to generate JSON, XML, and YAML outputs. For this purpose, it uses JacksonObjectMapper, which needs to walk over the class fields via reflection and perform heavy branching and intermediate object instantiation. On the contrary,log4j2-logstash-layoutparses the given template once and compiles a (mostly) garbage- and (to a certain extent) branching-free JSON generator employing JacksonJsonGenerator. -
Why is
log4j2-logstash-layoutis not totally garbage-free?-
Since
Throwable#getStackTrace()clones the originalStackTraceElement[], accesses to (and hence rendering) stack traces can never be garbage-free. -
Rendering of context data (that is, MDC) field values is garbage-free if the value is either
null, or of typeString,Short,Integer,Long, orbyte[].
-
-
How can one run the benchmark on his/her machine? After a fresh
mvn clean verifywithin the source directory, runlayout-benchmark/benchmark.py. -
What about thread-local allocations? Even though Log4j 2 exposes a
log4j2.enable.threadlocalsflag to toggle TLAs, neitherEcsLayout, nor Log4j 2JsonLayoutandGelfLayouthonor it. Historically,LogstashLayoutused to have TLAs taking thelog4j2.enable.threadlocalsflag into account. In version 0.18, we switched to simple memory-efficient object pools, though that incurred extra synchronization costs. Since version 0.22,LogstashLayoutswitched back to TLAs takinglog4j2.enable.threadlocalsinto account.
-
How can one enable thread-local allocations? For performance reasons, it is highly recommended to turn TLAs on. For this purpose, you need to make sure
log4j2.enable.threadlocals=trueandlog4j2.is.webapp=false. -
Is there a fat JAR of the plugin? Project also contains a
log4j2-logstash-layout-fatjarartifact which includes all its transitive dependencies in a separate shaded package (to avoid the JAR Hell) with the exception oflog4j-core, that you need to include separately. Fat JAR might come handy if you want to use this plugin along with certain applications, e.g., Elasticsearch 5.x and 6.x versions, which requires Log4j 2. -
Why do I get irrelevant stack traces from
exception[RootCause]:stackTrace[:text]directives?LogstashLayoutusesThrowable#printStackTrace(PrintWriter)andThrowable#getStackTrace()methods to resolve these directives. Under certain circumstances, these methods may throw exception as well. Put another way, trying to access the stack trace of an exception might cause another exception. In such a case,LogstashLayoutcontinues the directive resolution using the new exception and keeps on repeating until it manages to resolve a stack trace to at least provide some insight into the underlying cause.
- bakomchik
- chrissydee
- Daniel Lundsgaard Skovenborg
- Eric Schwartz
- Felix Barnsteiner
- Johann Schmitz
- John Ament
- Jonathan Guéhenneux
- justinsaliba
- Maurice Zeijen
- Michael K. Edwards
- Mikael Strand
- Rafa Gómez
- Yaroslav Skopets
Copyright © 2017-2020 Volkan Yazıcı
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "dependabot[bot]")
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent