Comments (8)
If I understand your comment right... Replacing
BBOS
withBAOS
would violate the garbage-free characteristics of the layout. Putting this detail aside, imagine that the JSON rendering is finalized and just before passing it to the appender you figured out that it violates the upper bound. Independent of whether the violation is measured onbyte
orchar
count, you cannot blindly truncate a complete JSON string anymore; hence throwing an exception is your only way out. (Please let me know if I interpret your feedback wrong.)Further, I am in the process of contributing this project into
log4j-core
(see [apache/logging-log4j2#335](apache/logging-log4j2#335)). This will deprecatelog4j2-logstash-layout
; though I will still address bugs, just like I do right now. There I will replacemaxByteCount
withmaxLength
, which will be based onchar
count and I believe to be more intuitive for users.
I am excited when I see you will contribute this project to log4j-core, but when you will finish it, I am expecting the log4j2's new layout.
from log4j2-logstash-layout.
If I understand your comment right... Replacing BBOS
with BAOS
would violate the garbage-free characteristics of the layout. Putting this detail aside, imagine that the JSON rendering is finalized and just before passing it to the appender you figured out that it violates the upper bound. Independent of whether the violation is measured on byte
or char
count, you cannot blindly truncate a complete JSON string anymore; hence throwing an exception is your only way out. (Please let me know if I interpret your feedback wrong.)
Further, I am in the process of contributing this project into log4j-core
(see apache/logging-log4j2#335). This will deprecate log4j2-logstash-layout
; though I will still address bugs, just like I do right now. There I will replace maxByteCount
with maxLength
, which will be based on char
count and I believe to be more intuitive for users.
from log4j2-logstash-layout.
I'm sorry, I did not consider garbage. And I missed the obvious fact that the OutputStream is for the whole object, not individual values, thus my rambling about maxStringLength.
Would it be possible to implement a retry mechanism with a lower maxStringLength to avoid message loss altogether? I.e., on BufferOverflowException, run the serialization again with half the maxStringLength until success.
from log4j2-logstash-layout.
I have my doubts if such a smart retry mechanism should really be implemented in this plugin. Would you mind sharing your use case, please? In an ideal world, application logs should not indeed be lost. That said, it is difficult come up with a policy for excessive loads. At bol.com, we records statistics for dropped logs. Would that also work for you? Thinking more about it... Maybe we can expose an MXbean to share such statistics in Log4j.
from log4j2-logstash-layout.
If we hit a database error we get a 22K stacktrace. We also set user input in the MDC which give us the opportunity to search in GrayLog for, e.g., error log messages logged during requests with a certain argument set to a certain value. Though we have turned off input logging for those of our POST methods that we know will receive large inputs, we could theoretically get large input in other methods (let's put validation aside for a minute). This is of course theoretical but ideally I wished I would not have to worry balancing maxStringLength
and maxByteCount
to get as much data as possible while not allocating too much each time we get the context (the environment is a web application and does thus not run with ENABLE_THREADLOCALS=true
).
I guess we could monitor our JBoss server log for BufferOverflowException. I'm not sure i understand your remark about MXBeans; do you use JMX to monitor this exception?
from log4j2-logstash-layout.
Had an epiphany while thinking about your problem and it resulted in something that I am really happy with. In JsonTemplateLayout
, the LogstashLayout
successor that I am trying to get into Log4j
core, I have removed the maxByteCount
configuration ‐ now there are just maxStringLength
and truncatedStringSuffix
. Technically, one can still come up with an explosive ObjectMessage
if ${json:message:json}
is used in the template, e.g., an array containing 1e6 strings each not exceeding maxStringLength
limit. But I believe that is not a surprising outcome given ${json:message:json}
is used. In conclusion, I think this setup has better ergonomics and causes less WTF moments for the user.
I am closing the issue. But let me know if you need further assistance.
from log4j2-logstash-layout.
I have removed the
maxByteCount
configuration ‐ now there are justmaxStringLength
andtruncatedStringSuffix
That sounds great! I suppose you refer to apache/logging-log4j2@15dd26c
So, as I understand it, we actually get a growing buffer then, because the StringBuilder
will raise its internal capacity as needed and never shrink it, even after you call setLength(0)
. Then we get a growing buffer and garbage freeness.
from log4j2-logstash-layout.
I have removed the
maxByteCount
configuration ‐ now there are justmaxStringLength
andtruncatedStringSuffix
That sounds great! I suppose you refer to apache/logging-log4j2@15dd26c
So, as I understand it, we actually get a growing buffer then, because theStringBuilder
will raise its internal capacity as needed and never shrink it, even after you callsetLength(0)
. Then we get a growing buffer and garbage freeness.
Correct.
from log4j2-logstash-layout.
Related Issues (20)
- BufferOverflowException get thrown for messages of a larger size HOT 7
- LogstashLayout with log4j2-elasticsearch appender HOT 1
- Introduce properties to globally set LogstashLayout.Builder defaults
- Nullpointer exceptions at LogstashLayoutSerializationContexts$StringTruncatingGeneratorDelegate HOT 1
- Any template ready for GCP? HOT 9
- Message lost if exceeding maxStringLength and emptyPropertyExclusionEnabled is true HOT 4
- Stacktraces larger than maxStringLength are not truncated HOT 7
- Failed serializing JSON: BufferOverflowException HOT 10
- How to output ThreadContext (ex -MDC fields) HOT 1
- Review request for Log4j JsonTemplateLayout HOT 7
- How exactly is MapMessage supposed to be handled? HOT 7
- Multiple template variables in single property lead to an exception HOT 2
- Fallback for json:map: fields for libraries HOT 1
- NoClassDefFoundError: logstash.layout.resolver.TemplateResolvers (initialization failure) HOT 1
- Remove max byte count cap? HOT 1
- Reference whole MapMessage - and not just keys HOT 13
- Compiling with JDK 8 results in java.lang.NoSuchMethodError: java.nio.ByteBuffer.flip()Ljava/nio/ByteBuffer; HOT 4
- Marker lookup subsitutes literal string when marker not available HOT 1
- BufferOverflowException occurs intermittently when sending data via socket appender HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from log4j2-logstash-layout.