Giter Site home page Giter Site logo

sumologic-log4j2-appender's Introduction

Build Status codecov.io

sumologic-log4j2-appender

A Log4j2 appender that sends straight to Sumo Logic.

For the Logback appender, please see https://github.com/SumoLogic/sumologic-logback-appender

Installation

The library can be added to your project using Maven Central by adding the following dependency to a POM file:

<dependency>
    <groupId>com.sumologic.plugins.log4j</groupId>
    <artifactId>sumologic-log4j2-appender</artifactId>
    <version>2.2.0</version>
</dependency>

Usage

Set up HTTP Hosted Collector Source in Sumo Logic

Follow these instructions for setting up an HTTP Source in Sumo Logic.

Log4J XML Configuration

Be sure to replace [collector-url] with the URL after creating an HTTP Hosted Collector Source in Sumo Logic.

log4j2.xml:

<?xml version="1.0" encoding="UTF-8"?>
<Configuration>
    <Appenders>
        <SumoLogicAppender
                name="SumoAppender"
                url="[collector-url]">
            <PatternLayout pattern="%d{yyyy-MM-dd HH:mm:ss,SSS Z} [%t] %-5p %c - %m%n" />
        </SumoLogicAppender>
    </Appenders>
    <Loggers>
        <Root level="all" additivity="false">
            <AppenderRef ref="SumoAppender" />
        </Root>
    </Loggers>
</Configuration>

Note: We recommending starting your PatternLayout pattern with a date and time such as %d{yyyy-MM-dd HH:mm:ss,SSS Z} for two reasons:

  1. Having a consistent prefix that starts every message is necessary for multiline boundary detection to learn the message prefix needed to group mutiline messages, such as stack traces.
  2. Sumo only supports certain time formats, and accidentally using an invalid time format could cause message time discrepancies.

Parameters

Parameter Required? Default Value Description
name Yes Name used to register Log4j Appender
url Yes HTTP collection endpoint URL
sourceName No "Http Input" Source name to appear when searching on Sumo Logic by _sourceName
sourceHost No Client IP Address Source host to appear when searching on Sumo Logic by _sourceHost
sourceCategory No "Http Input" Source category to appear when searching on Sumo Logic by _sourceCategory
proxyHost No Proxy host IP address
proxyPort No Proxy host port number
proxyAuth No For basic authentication proxy, set to "basic". For NTLM authentication proxy, set to "ntlm". For no authentication proxy, do not specify.
proxyUser No Proxy host username for basic and NTLM authentication. For no authentication proxy, do not specify.
proxyPassword No Proxy host password for basic and NTLM authentication. For no authentication proxy, do not specify.
proxyDomain No Proxy host domain name for NTLM authentication only
retryInterval No 10000 Retry interval (in ms) if a request fails
maxNumberOfRetries No -1 Maximum number of retries before a message is dropped. Negative values represent no limit on retries.
connectionTimeout No 1000 Timeout (in ms) for connection
socketTimeout No 60000 Timeout (in ms) for a socket
messagesPerRequest No 100 Number of messages needed to be in the queue before flushing
maxFlushInterval No 10000 Maximum interval (in ms) between flushes
flushingAccuracy No 250 How often (in ms) that the flushing thread checks the message queue
maxQueueSizeBytes No 1000000 Maximum capacity (in bytes) of the message queue
flushAllBeforeStopping No false Flush all messages before stopping regardless of flushingAccuracy
retryableHttpCodeRegex No ^5.* Regular expression specifying which HTTP error code(s) should be retried during sending. By default, all 5xx error codes will be retried.

Example with Optional Parameters

log4j2.xml:

<?xml version="1.0" encoding="UTF-8"?>
<Configuration>
    <Appenders>
        <SumoLogicAppender
                name="SumoAppender"
                url="[collector-url]"
                flushAllBeforeStopping="true"
                sourceHost="Appender-$${env:HOSTNAME}"
                sourceCategory="${sys:appName}"
                proxyHost="1.2.3.4"
                proxyPort="3128">
            <PatternLayout pattern="%d{yyyy-MM-dd HH:mm:ss,SSS Z} [%t] %-5p %c - %m%n" />
        </SumoLogicAppender>
    </Appenders>
    <Loggers>
        <Root level="all" additivity="false">
            <AppenderRef ref="SumoAppender" />
        </Root>
    </Loggers>
</Configuration>

TLS 1.2 Requirement

Sumo Logic only accepts connections from clients using TLS version 1.2 or greater. To utilize the content of this repo, ensure that it's running in an execution environment that is configured to use TLS 1.2 or greater.

Development

To compile the plugin:

  • Run "mvn clean package" on the pom.xml in the main level of this project.
  • To test running a locally built JAR file, you may need to manually add the following dependencies to your project:
    <dependencies>
        <dependency>
            <groupId>org.apache.logging.log4j</groupId>
            <artifactId>log4j-core</artifactId>
            <version>2.19.0</version>
        </dependency>

        <dependency>
            <groupId>org.apache.httpcomponents</groupId>
            <artifactId>httpclient</artifactId>
            <version>4.5.13</version>
        </dependency>
    </dependencies>

License

The Sumo Logic Log4j 2 Appender is published under the Apache Software License, Version 2.0. Please visit http://www.apache.org/licenses/LICENSE-2.0.txt for details.

sumologic-log4j2-appender's People

Contributors

delphym avatar dependabot[bot] avatar drcrallen avatar duchatran avatar eddie-sumo avatar germanborbolla avatar mccartney avatar mzochowskisumo avatar perk-sumo avatar raphisuter avatar rvmiller89 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

sumologic-log4j2-appender's Issues

Various changes

FYI, we did a refactor / rewrite of some components to fit our needs and put the results at https://github.com/metamx/sumologic-log4j2-appender

The biggest changes are in metamx#1 , but we also moved the versioning to a ${log4j.version}.${patch} scheme instead of the one in the upstream branch which made it impossible to tell which log4j2 version the plugin was built for. The more recent versions of log4j2 induce a lot of plugin changes that make it hard to keep "how to write a plugin" compatible across arbitrary legacy versions (i.e. using log4j2 2.8.x plugin definitions in a 2.5 impl)

I don't know if any of these are interesting, but it is apache licensed in our branch so they can be pulled upstream.

A few notable changes are:

  • Improved concurrency
  • Exponential backoff with jitter on batch uploads
  • a SumoJsonFormat which produces a Json format that is mostly friendly with SumoLogic's json parsing. (JSON objects over 64kb get split by SumoLogic and do not parse correctly)
  • Change versioning semantics

Flush all pending messages on shutdown

I have noticed that on shutdown of the JVM - it doesn't look like all the messages are flushed out to Sumo.
I have a short-running batch process (running in AWS Batch) which terminates (with System.exit) at the end of it's processing. I am noticing that not all the messages logged are being sent to Sumo - and i think it is because there are still messages in the buffer which are not sent on shutdown.

I have looked at the code and can't see a way to explicitly flush messages (which would work for me). It looks like the best I can do is to set the messagesPerRequest to a low number (10ish) and the maxFlushInterval to a smaller number in order to make sure messages are flushed more often. This doesn't guarantee that all the messages are sent however.

Maybe the stop() method on the SumoLogicAppender could flush all pending messages prior to calling sender.close() and flusher.stop()?

Or - add a shutdown hook to send messages before the JVM shuts down?

inclusion of log4j2.xml causes issues

Including a log4j2.xml configuration file can cause issues when alternate configuration methods are used. Is there a reason that the file is included in the main sources root vs just for tests?

Support for category

Hi.

It would be nice, if the category could be configured with the appender config. This would allow having only one endpoint for different environments like dev, stg etc. This could be simply set on the appender using a log4j2 env variable or something.

Now I have to create one collector per environment an set the category there and have to pass the different collector endpoints per environment.

Testing is system dependent

inside the test class SumoLogicAppenderTest.testMessagesWithMetadata on line 81 the linux line separator is hardcoded. If you run the tests in windows, the test will fail.

  for(MaterializedHttpRequest request: handler.getExchanges()) {
            for (String line : request.getBody().split("\n")) {
                // Strip timestamp
                int mainStart = line.indexOf("[main]");
                String trimmed = line.substring(mainStart);
                actual.append(trimmed + "\n");
            }
        }

Inconsistent logging of Mule logger messages via log4j2 appender

after raising this issue with official Sumologic support i was told that because the log4j2 appender was OSS that they did not (officially) support it and to try raising the issue here.

i'll provide an overview first, but can provide more details as necessary (i'm also not sure how much these repos are supported by Sumologic or whoever else).

in short, we are using the log4j2 appender to connect to a Sumo HTTP collector via URL, as follows, in a Mulesoft project

  <?xml version="1.0" encoding="utf-8"?> 
  <Configuration status="info" name="sumologic" packages="com.sumologic.log4j"> 
    <Appenders> 
      <RollingFile name="file" fileName="${sys:mule.home}${sys:file.separator}logs${sys:file.separator}test-logging.log" 
        filePattern="${sys:mule.home}${sys:file.separator}logs${sys:file.separator}test-logging-%i.log"> 
        <PatternLayout pattern="%d [%t] %-5p %c - %m%n" /> 
        <SizeBasedTriggeringPolicy size="10 MB" /> 
        <DefaultRolloverStrategy max="10"/> 
      </RollingFile> 
      <!-- sumologic appender / collector url --> 
      <SumoLogicAppender name="SumoAppender" url="${sys:sumologic.appender.url}"> 
        <PatternLayout pattern="%d{yyyy-MM-dd HH:mm:ss,SSS Z} [%t] %-5p %c - %m%n" /> 
        </SumoLogicAppender> 
      </Appenders> 
      ...
      <Loggers> 
        ... 
        <AsyncRoot level="INFO"> 
          <AppenderRef ref="file" /> 
          <AppenderRef ref="SumoAppender" /> 
        </AsyncRoot> 
      </Loggers> 
  </Configuration>

and i find that while i'm seeing Mule logging messages appear under the category on Sumologic, eg:

  26  10/07/2018
  14:48:18.295 +1000  
  2018-07-10 14:48:18,295 +1000 [main] INFO  org.mule.processor.SedaStageLifecycleManager - Initialising service: test-sumologic-by-logger-test.stage1
  Host: 103.199.119.138  Name: Http Input  Category: tmp/monkey/business-two 
  27  10/07/2018
  14:48:18.288 +1000  
  2018-07-10 14:48:18,288 +1000 [main] INFO  org.mule.exception.DefaultMessagingExceptionStrategy - Initialising exception listener: org.mule.exception.DefaultMessagingExceptionStrategy@2f95653f
  Host: 103.199.119.138  Name: Http Input  Category: tmp/monkey/business-two 
  28  10/07/2018
  14:48:18.288 +1000  
  2018-07-10 14:48:18,288 +1000 [main] INFO  org.mule.construct.FlowConstructLifecycleManager - Initialising flow: test-sumologic-by-logger-test
  Host: 103.199.119.138  Name: Http Input  Category: tmp/monkey/business-two 
  29  10/07/2018
  14:48:18.247 +1000  
  2018-07-10 14:48:18,247 +1000 [main] INFO  org.mule.processor.SedaStageLifecycleManager - Initialising service: test-sumologic.stage1
  Host: 103.199.119.138  Name: Http Input  Category: tmp/monkey/business-two 
  30  10/07/2018
  14:48:18.198 +1000  
  2018-07-10 14:48:18,198 +1000 [main] INFO  org.mule.exception.DefaultMessagingExceptionStrategy - Initialising exception listener: org.mule.exception.DefaultMessagingExceptionStrategy@d653e41
  Host: 103.199.119.138  Name: Http Input  Category: tmp/monkey/business-two 
  31  10/07/2018
  14:48:18.197 +1000  
  2018-07-10 14:48:18,197 +1000 [main] INFO  org.mule.construct.FlowConstructLifecycleManager - Initialising flow: test-sumologic
  Host: 103.199.119.138  Name: Http Input  Category: tmp/monkey/business-two 
  32  10/07/2018
  14:48:18.181 +1000  
  2018-07-10 14:48:18,181 +1000 [main] INFO  org.mule.lifecycle.AbstractLifecycleManager - Initialising model: _muleSystemModel
  Host: 103.199.119.138  Name: Http Input  Category: tmp/monkey/business-two 
  33  10/07/2018
  14:48:17.893 +1000  
  2018-07-10 14:48:17,893 +1000 [main] INFO  org.mule.util.journal.TransactionJournal - Using files for tx logs /home/vagrant/AnypointStudio/workspace-6-3/test-logging/./.mule/queue-xa-tx-log/tx1.log and /home/vagrant/AnypointStudio/workspace-6-3/test-logging/./.mule/queue-xa-tx-log/tx2.log
  Host: 103.199.119.138  Name: Http Input  Category: tmp/monkey/business-two 
  34  10/07/2018
  14:48:17.584 +1000  
  2018-07-10 14:48:17,584 +1000 [main] INFO  org.mule.util.journal.TransactionJournal - Using files for tx logs /home/vagrant/AnypointStudio/workspace-6-3/test-logging/./.mule/queue-tx-log/tx1.log and /home/vagrant/AnypointStudio/workspace-6-3/test-logging/./.mule/queue-tx-log/tx2.log
  Host: 103.199.119.138  Name: Http Input  Category: tmp/monkey/business-two 
  35  10/07/2018
  14:48:17.090 +1000  
  2018-07-10 14:48:17,090 +1000 [main] WARN  org.mule.module.ognl.expression.OgnlExpressionEvaluator - OGNL module is deprecated and will be removed in Mule 4.0. Use MEL expressions instead.
  Host: 103.199.119.138  Name: Http Input  Category: tmp/monkey/business-two 

that the messages i'm logging in the actual Mule flow, via the Mule standard logging component, are not appearing, even though they use the same logging mechanism (log4j2) as Mule itself

  psc>>>> log-sumologic-by-logger-sub-start
  psc>>>> hello, world from mule...
  psc>>>> log-sumologic-by-logger-sub-end

sometimes we see the first message, but not the next two and, most recently, all the messages fail to appear; note that all the messages are visible in the Mule / Anypoint Studio console.

this has been happening repeatedly in past days (tues-wed, 10-11Jul18 AEST) and in recent weeks, while, a month or two back, it all seemed to work consistently.

~p

Turning off "SumoBufferFlusherThread" log messages

Hi,
I am seeing a large number of DEBUG messages of similar type to below which are filling up in Sumo Logic pretty much accompanying all logs getting pushed.

DEBUG 2020-04-21 13:44:18,818 [SumoBufferFlusherThread] : Tue Apr 21 13:44:18 AEST 2020 - Flushing and sending out 147 messages (0 messages left)
DEBUG 2020-04-21 13:44:18,818 [SumoBufferFlusherThread] : Sending out data

Is there a way to turn this log messages off from a setting/config in log4j2.xml?

Thanks.

Unable to send log messages to SumoLogic from Eclipse

Hi,
I'm using the SumoLogic appender (sumologic-log4j2-appender) in log4j2.xml in a small test application that is running on Eclipse. I am sending a log info and a error messages and they don't show up at all in SumoLogic. But the console log says that :"main DEBUG Sending message to Sumo: 2018-10-30 15:17:41,730 -0600 [main] INFO com.sumologic.log4j.SumoLogicAppender - Hello Document Manager from Logback!"
Is there some thing I need to set up on the appender to successfully send messages?

Thanks,

DMLogger.zip

Cannot compile project on fresh clone

When running mvn clean package on a fresh clone I get the following error:

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.8.1:compile (default-compile) on project sumologic-log4j2-appender: Compilation failure: Compilation failure: 
[ERROR] Source option 6 is no longer supported. Use 7 or later.
[ERROR] Target option 6 is no longer supported. Use 7 or later.
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException

Large messages do not show up in SumoLogic

Hi,

I'm using the SumoLogic appender in log4j.xml in a Mule application. I am logging some large messages (a java ArrayList with 10,000 elements) and they don't show up at all in SumoLogic.
Is there some setting that can be tweaked on the appender to successfully send large messages?

Thanks,
Elza

Failed to load SumoLogicAppender when the app is a fat jar

It works if I start the app from IDE, but if I build the app into a fat jar and run it, it doesn't work.
I can confirm the com.sumologic.log4j.SumoLogicAppender.class is included in the fat jar.

It got this error ERROR Error processing element SumoLogicAppender ([Appenders: null]): CLASS_NOT_FOUND

<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="INFO">
    <Appenders>
        <SumoLogicAppender
                name="SumoAppender"
                url="xxxx"
                sourceName="mySource"
                sourceHost="myHost2"
                sourceCategory="myCategory"
                messagesPerRequest="1000"
                maxFlushInterval="100"
                flushingAccuracy="10">
            <JsonLayout compact="true"
                        eventEol="true"
                        properties="true">
                <KeyValuePair key="additionalField1" value="constant value"/>
            </JsonLayout>
        </SumoLogicAppender>
        <Console name="console2" target="SYSTEM_OUT">
            <JsonLayout compact="true"
                        eventEol="true"
                        properties="true">
                <KeyValuePair key="additionalField1" value="constant value"/>
            </JsonLayout>

        </Console>

    </Appenders>
    <Loggers>
        <Root level="INFO" additivity="false">
            <AppenderRef ref="console2"/>
            <AppenderRef ref="SumoAppender"/>
        </Root>
    </Loggers>
</Configuration>

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.