netflix/blitz4j

Logging framework for fast asynchronous logging

Blitz4j

Blitz4j is a logging framework built on top of log4j to reduce contention and enable highly scalable logging without affecting application performance characteristics.

At Netflix, Blitz4j is used to log billions of events for monitoring, business intelligence reporting, debugging and other purposes. Blitz4j overcomes traditional log4j bottlenecks and comes built with a highly scalable and customizable asynchronous framework. Blitz4j comes with the ability to convert the existing log4j appenders to use the asynchronous model without changing the existing log4j configurations.

Blitz4j uses archaius (http://www.github.com/Netflix/archaius) to reconfigure log4j dynamically and servo (http://www.github.com/Netflix/servo) to track the performance metrics regarding logging. Blitz4j also tries to mitigate data loss and provides a way to summarize the log information during log storms.

BUILD

./gradlew clean build

Support

[Blitz4j Google Group] (https://groups.google.com/forum/?fromgroups#!forum/netflix_blitz4j)

Documentation

Please see [wiki] (https://github.com/Netflix/blitz4j/wiki) for detailed documentation.

blitz4j's Issues

asyn leads to loss of the logs contens

i create a spring boot project, based on the spring mvc. every request, print 1 logs. i send request with jmeter, 100 threads, cycle 1000times every thread。 i should get 100*1000 = 100000 logs, bug i get 86000logs. if i use the synchronization mode，i can get 100000logs。 why？

did blitz4j compare with logback

benchmark needed

The supplied unit tests don't work

Firstly, the unit tests are not properly included into the Gradle configuration.
Once that it done using the following build.gradle block:

sourceSets {
    test {
        java {
            srcDirs  = ['test/main/java']
        }
    }
}

... the tests run and fail. Both of them.

From the output that they generate at least these things are obvious:

a) slf4j-simple dependency is missing (the infamous SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". error)
b) PatternLayout settings are obviously ignored since the pattern of the printed messages is the log4j default
c) Reconfiguration attempt of the testReconfiguration unit test gets ignored

The above is true for both the current v1.13 github code and v1.11 distribution from Maven.
I've tried them both and I could not make any of them work.

.travis.yml: The 'sudo' tag is now deprecated in Travis CI

Travis are now recommending removing the sudo tag.

"If you currently specify sudo: false in your .travis.yml, we recommend removing that configuration"

Optimize for log4j2

Hello,

it is possible to make blitz4j compatible with log4j2 ?

I try to start Eureka with log4j2 and get the error:

thanks in advance
Marcel

Exception in thread "main" org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.springframework.cloud.netflix.eureka.server.EurekaServerInitializerConfiguration': Invocation of init method failed; nested exception is java.lang.VerifyError: class com.netflix.blitz4j.NFLockFreeLogger overrides final method getEffectiveLevel.()Lorg/apache/log4j/Level;
    at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:136)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:408)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1558)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:539)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:476)
    at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:303)
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:299)
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:755)
    at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:757)
    at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:480)
    at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.refresh(EmbeddedWebApplicationContext.java:118)
    at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:686)
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:320)
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:957)
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:946)
    at eurekademo.EurekaApplication.main(EurekaApplication.java:19)
Caused by: java.lang.VerifyError: class com.netflix.blitz4j.NFLockFreeLogger overrides final method getEffectiveLevel.()Lorg/apache/log4j/Level;
    at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
    at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
    at java.net.URLClassLoader.defineClass(URLClassLoader.java:455)
    at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:367)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
    at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
    at java.net.URLClassLoader.defineClass(URLClassLoader.java:455)
    at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:367)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    at org.springframework.cloud.netflix.eureka.server.EurekaServerInitializerConfiguration.initLogging(EurekaServerInitializerConfiguration.java:103)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:483)
    at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:349)
    at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:300)
    at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:133)
    ... 17 more

Question: Advantage blitz4j vs log4j2 async logger

I haven't found a good comparison of blitz4j vs the new log4j2 async logger.

Can somebody give me a short answer if it is worth to change to blitz4j if I already have log4j2 in place.

I know blitz4j provides more features like handling log storms which comes very handy but from a scalability (less contention etc) pov I dont see a real reason (in fact I think log4j2 performs better because of the Disruptor).
Is this a reasonable statement?

question - migrate from log4j to blitz4j

hi,
I was hoping if someone could help me with this.
specifically regarding the log4j.xml file

thanks!

Location information is lost without summarizeOverflow turned on

LoggingConfiguration.java doesn't follow fallback logic on Malformed URLs

The "Default Initialization Procedure" section of the https://logging.apache.org/log4j/1.2/manual.html page says that:

"If the resource variable cannot be converted to a URL, for example due to a MalformedURLException, then search for the resource from the classpath by calling org.apache.log4j.helpers.Loader.getResource(resource, Logger.class) which returns a URL. Note that the string "log4j.properties" constitutes a malformed URL."

However, the way the LoggingConfiguration configure method is written, it doesn't do that. If the log4jConfigurationFile (as provided by a -Dlog4j.configuration property) is not null (for example, "mylog4j.properties") it will call loadLog4jConfigurationFile which tries to create a URL and which throws a RuntimeException for the Malformed URL, instead of falling back on looking for the resource in the classpath, which it only does if the log4jConfigurationFile is null.

This is a big miss in the configuration.

Question: Can you show evidence to support claimed "enable highly scalable logging without affecting application performance characteristics"?

I am all for improved logging performance, but have not been able to find supporting evidence to the claims. Can you please point me at the right direction? some before/after case study? reproducible results?
Thanks