Comments (9)
Processing payloads was sort of cryptic to me given my lack of knowledge about its workings. I had a better visual with the StackdriverJsonLayout.java
of spring-cloud-gcp-logging
project as you pointed in the Log4j mailing list. Let's see which of the fields are resolvable by available LogstashLayout
directives and which might necessitate new ones:
if (this.includeMDC) {
event.getMDCPropertyMap().forEach((key, value) -> {
if (!FILTERED_MDC_FIELDS.contains(key)) {
map.put(key, value);
}
});
}
mdc
directive perfectly suffices here. But in its current form, there is no way to unwrap it at the top level.
if (this.includeTimestamp) {
map.put(StackdriverTraceConstants.TIMESTAMP_SECONDS_ATTRIBUTE,
TimeUnit.MILLISECONDS.toSeconds(event.getTimeStamp()));
map.put(StackdriverTraceConstants.TIMESTAMP_NANOS_ATTRIBUTE,
TimeUnit.MILLISECONDS.toNanos(event.getTimeStamp() % 1_000));
}
Here I'd rather go with "timestamp": "${json:timestamp}"
, which is allowed by the GCP documentation. The only catch here is to make sure that timeZoneId
is set to UTC
.
add(StackdriverTraceConstants.SEVERITY_ATTRIBUTE, this.includeLevel,
String.valueOf(event.getLevel()), map);
"severity": "${json:level}"
(or level:severity
?)
add(JsonLayout.THREAD_ATTR_NAME, this.includeThreadName, event.getThreadName(), map);
"thread": "${json:thread:name}"
add(JsonLayout.LOGGER_ATTR_NAME, this.includeLoggerName, event.getLoggerName(), map);
"logger": "${json:logger:name}"
if (this.includeFormattedMessage) {
String message = event.getFormattedMessage();
if (this.includeExceptionInMessage) {
IThrowableProxy throwableProxy = event.getThrowableProxy();
if (throwableProxy != null) {
String stackTrace = getThrowableProxyConverter().convert(event);
if (stackTrace != null && !stackTrace.equals("")) {
message += "\n" + stackTrace;
}
}
}
map.put(JsonLayout.FORMATTED_MESSAGE_ATTR_NAME, message);
}
This is interesting. When includeFormattedMessage
is true
, message
field is set to formattedMessage + '\n' + stackTrace
, if there is an exception; otherwise, formattedMessage
itself. I guess we don't need this. Continue reading...
add(JsonLayout.MESSAGE_ATTR_NAME, this.includeMessage, event.getMessage(), map);
"message": "${json:message}"
add(JsonLayout.CONTEXT_ATTR_NAME, this.includeContextName, event.getLoggerContextVO().getName(), map);
We don't have an equivalent for this in LogstashLayout
. It feels like we can just ignore this.
addThrowableInfo(JsonLayout.EXCEPTION_ATTR_NAME, this.includeException, event, map);
"exception": "${json:exception:stackTrace:text}"
addTraceId(event, map);
which translates to
protected String formatTraceId(final String traceId) {
// Trace IDs are either 64-bit or 128-bit, which is 16-digit hex, or 32-digit hex.
// If traceId is 64-bit (16-digit hex), then we need to prepend 0's to make a 32-digit hex.
if (traceId != null && traceId.length() == 16) {
return "0000000000000000" + traceId;
}
return traceId;
}
private void addTraceId(ILoggingEvent event, Map<String, Object> map) {
if (!this.includeTraceId) {
return;
}
String traceId =
event.getMDCPropertyMap().get(StackdriverTraceConstants.MDC_FIELD_TRACE_ID);
if (traceId == null) {
traceId = TraceIdLoggingEnhancer.getCurrentTraceId();
}
if (!StringUtils.isEmpty(traceId)
&& !StringUtils.isEmpty(this.projectId)
&& !this.projectId.endsWith("_IS_UNDEFINED")) {
traceId = StackdriverTraceConstants.composeFullTraceName(
this.projectId, formatTraceId(traceId));
}
add(StackdriverTraceConstants.TRACE_ID_ATTRIBUTE, this.includeTraceId, traceId, map);
}
That's quite some business logic going here. An innocent attempt can be "logging.googleapis.com/trace": "${json:mdc:X-B3-TraceId}"
, but I doubt if this can cover corner cases (e.g., 32-digit hex issue) mentioned in the code above.
add(StackdriverTraceConstants.SPAN_ID_ATTRIBUTE, this.includeSpanId,
event.getMDCPropertyMap().get(StackdriverTraceConstants.MDC_FIELD_SPAN_ID), map);
"logging.googleapis.com/spanId": "${json:mdc:X-B3-SpanId}"
if (this.serviceContext != null) {
map.put(StackdriverTraceConstants.SERVICE_CONTEXT_ATTRIBUTE, this.serviceContext);
}
I don't have a solution here except dragging serviceContext
via an MDC field.
if (this.customJson != null && !this.customJson.isEmpty()) {
for (Map.Entry<String, Object> entry : this.customJson.entrySet()) {
map.putIfAbsent(entry.getKey(), entry.getValue());
}
}
This can be achieved by either modifying the template or providing eventTemplateAdditionalFields
.
In the light of these findings, at a first glance, the following template might do the trick:
{
"mdc1": "${json:mdc:key1}",
"mdc2": "${json:mdc:key2}",
"mdc3": "${json:mdc:key3}",
"timestamp": "${json:timestamp}",
"severity": "${json:level}",
"thread": "${json:thread:name}",
"logger": "${json:logger:name}",
"message": "${json:message}",
"exception": "${json:exception:stackTrace:text}",
"logging.googleapis.com/trace": "${json:mdc:X-B3-TraceId}",
"serviceContext": "${json:mdc:serviceContext}"
}
@ilgrosso, would you mind giving this a try, please? Any other feedback and/or comments are also welcome.
from log4j2-logstash-layout.
The above looks great, thanks!
I'll give it a try early next week and report here.
from log4j2-logstash-layout.
Thanks for you help @vy , worked like a charm.
Important note is also to set prettyPrintEnabled="false"
.
from log4j2-logstash-layout.
@ilgrosso, before closing the issue, it might be good that we add this as a FAQ to README. Would you mind sharing how did you exactly solve your problem, please? The template JSON, Java code, prepared MDC fields, etc. The more details you provide, the more helpful it will be.
from log4j2-logstash-layout.
I have no MDC fields (yet?), no Java code to add.
Just added the Maven dependency, grabbed your JSON template as above and defined appenders like as follows:
<Console name="mainFile" target="SYSTEM_OUT">
<LogstashLayout dateTimeFormatPattern="yyyy-MM-dd'T'HH:mm:ss.SSSZZZ"
timeZoneId="UTC"
eventTemplateUri="classpath:logstashGCP.json"
prettyPrintEnabled="false"
stackTraceEnabled="true"/>
</Console>
from log4j2-logstash-layout.
How did you deal with logging.googleapis.com/trace
, serviceContext
, etc. fields? In my JSON template I was expecting them to be injected into MDC.
from log4j2-logstash-layout.
Ah sorry for not reporting: at the moment I don't need neither logging.googleapis.com/trace
nor serviceContext
- and they are both reported as null
in the actual statements, as expected.
from log4j2-logstash-layout.
How can I get custom field say version and it should read the value from other properties file or json file . We don't want to hard code it and wanted to dynamically populate and don't want to put it in mdc.
from log4j2-logstash-layout.
How can I get custom field say version and it should read the value from other properties file or json file . We don't want to hard code it and wanted to dynamically populate and don't want to put it in mdc.
First, I would strongly advise you to switch from LogstashLayout
provided by log4j2-logstash-layout
artifact to JsonTemplateLayout
provided by the official Log4j project in log4j-layout-template-json
artifact. LogstashLayout
is not developed anymore and JsonTemplateLayout
is the successor.
You can define the values either as JVM properties or environment variables. Then you can access these using lookups. This is explained (and demonstrated) in detail in the documentation of both projects.
from log4j2-logstash-layout.
Related Issues (20)
- BufferOverflowException get thrown for messages of a larger size HOT 7
- LogstashLayout with log4j2-elasticsearch appender HOT 1
- Introduce properties to globally set LogstashLayout.Builder defaults
- Nullpointer exceptions at LogstashLayoutSerializationContexts$StringTruncatingGeneratorDelegate HOT 1
- Message lost if exceeding maxStringLength and emptyPropertyExclusionEnabled is true HOT 4
- Use ByteArrayOutputStream instead of ByteBufferOutputStream to avoid message loss HOT 8
- Stacktraces larger than maxStringLength are not truncated HOT 7
- Failed serializing JSON: BufferOverflowException HOT 10
- How to output ThreadContext (ex -MDC fields) HOT 1
- Review request for Log4j JsonTemplateLayout HOT 7
- How exactly is MapMessage supposed to be handled? HOT 7
- Multiple template variables in single property lead to an exception HOT 2
- Fallback for json:map: fields for libraries HOT 1
- NoClassDefFoundError: logstash.layout.resolver.TemplateResolvers (initialization failure) HOT 1
- Remove max byte count cap? HOT 1
- Reference whole MapMessage - and not just keys HOT 13
- Compiling with JDK 8 results in java.lang.NoSuchMethodError: java.nio.ByteBuffer.flip()Ljava/nio/ByteBuffer; HOT 4
- Marker lookup subsitutes literal string when marker not available HOT 1
- BufferOverflowException occurs intermittently when sending data via socket appender HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from log4j2-logstash-layout.