urielha/log4stash-old

Module to Log log4net Messages to ElasticSearch

log4stash-old's Issues

The operation has timed out

Hi i am using latest version of log4stash and i am getting this error...

2015-11-21 02:21:06,114 [37] FATAL CommunicationErrorWriteToFile - Program Unhandled exception
System.Net.WebException: The operation has timed out
at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
at log4net.ElasticSearch.WebElasticClient.FinishGetResponse(IAsyncResult result) in d:\uriel\Programing\C#\log4stash\src\log4net.ElasticSearch\ElasticClient.cs:line 110
at System.Net.LazyAsyncResult.Complete(IntPtr userToken)
at System.Net.ContextAwareResult.CompleteCallback(Object state)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Net.ContextAwareResult.Complete(IntPtr userToken)
at System.Net.LazyAsyncResult.ProtectedInvokeCallback(Object result, IntPtr userToken)
at System.Net.HttpWebRequest.Abort(Exception exception, Int32 abortState)
at System.Net.HttpWebRequest.AbortWrapper(Object context)
at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
at System.Threading.ThreadPoolWorkQueue.Dispatch()
at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback()

Message and exception provided via LoggingEvent created from LoggingEventData is not sent to Elastic Search

We have a remote system that submits information to be logged to our server. This information includes the message and sometimes exception data.

On our server we then log the message using log4net like this:

var eventData = new LoggingEventData {
    LoggerName = diagnostic.loggerName,
    Message = diagnostic.message",
    ExceptionString = diagnostic.maybeExceptionString,
    TimeStamp = diagnostic.timeStamp,
    // etc...
};

var loggingEvent = new LoggingEvent(eventData);

loggingEvent.Fix = FixFlags.All;

ILogger logger = ... // Get ILogger
logger.Log(loggingEvent)

Everything works as expected when using a log4net FileAppender, but when submitted to Elastic Search using log4stash the message and exception are not sent.

I tracked this down to the ElasticSearchAppender implementation where it gets the message from loggingEvent.MessageObject.ToString() and the exception from loggingEvent.ExceptionObject.ToString(). A LoggingEvent created from LoggingEventData does not have values for these properties.

It seems like the correct place to get this information is loggingEvent.RenderedMessage and loggingEvent.GetExceptionString().

Note that GetExceptionString() returns an empty string rather than null if no exception information is provided, so you'll probably need to using string.IsNullOrEmpty() when checking whether or not there's exception information to send.

Error with properties with same name

Ran into this error when trying to use the appender. Background is that we are using our own built logging framework built upon Log4Net. What we have there is basically some extra properties.

System.ArgumentException: An item with the same key has already been added.
at System.Collections.Generic.Dictionary`2.Insert(TKey key, TValue value, Boolean add)
at log4net.ElasticSearch.ElasticSearchAppender.CreateLogEvent(LoggingEvent loggingEvent) in d:\uriel\Programing\C#\log4stash\src\log4net.ElasticSearch\ElasticSearchAppender.cs:line 235
at log4net.ElasticSearch.ElasticSearchAppender.Append(LoggingEvent loggingEvent) in d:\uriel\Programing\C#\log4stash\src\log4net.ElasticSearch\ElasticSearchAppender.cs:line 112
at log4net.Appender.AppenderSkeleton.DoAppend(LoggingEvent loggingEvent)�

Configure multiple elastic nodes

It would be nice if it was possible to define multiple ES nodes in the config, as a means of targeting a multi-node cluster to handle a single node failure.

Test against new Elastic version

Test log4stash against new versions of Elasticsearch.

dynamic indexname

I usually configure the IndexName with a fixed part (the ImndexName prefix) + the rolling date part.
It worked smoothly.
Now I need something more complex, I need to change the IndexName prefix from "the fixed part" to "a dynamic part" based on a message field like %logger for instance. I tried using both %logger and %{logger} but anytime it has been literally copied as part of the final IndexName and not evaluated.

Performance Improvements to ElasticClient

Hi,

I have read the code of ElasticClient, and I have some tips to improve performance of indexing:

• You are using .NET JavascriptSerializer, which is horribly slow and you are recreating it's instance every serialize (which clears it's internal cache), I recommend using Jil
• Instead of using WebRequest (and it's ugly Begin/End async pattern), you can use HttpClient and execute PutAsync/PostAsync - from my tests, It handles concurrency pretty well.

I will submit a pull request on the weekend If I will have time.

Stopped working after upgrading Elastic to 2.2

json message double encoding

I have a custom object that is serialized for use in a log4net fileappender. (I'm using a Newtonsoft to serialize my object)
This object is passed in as Log.(customobject) which gets persisted correctly to the txt log file.

I also have the log4stash (elasticsearch appender) setup which sends the object to my ES instance.

The problem is, with my custom object, it appends to the text file correctly , however, because the object is already serialized, as it is transferred to ES, it is serialized again and my json is double escaped.

EG.

....
""eventType"": "Location",
""eventTypeAction"": "CheckOut",
""message"": "Checkout was approved",
.....

Which screws with the mappings of my index.

If I remove my ToString() implementation for serializing my object, it is only serialized once on the way to ES and the json is inserted correctly, however my filelog appender only receives the object , un-serialized , and thus only prints the class name.

Is there a way to disable serialization into ES if this is already done? or is this a log4net issue.

Thanks

Custom log4net values get submitted as { "Count": 1 }

We log custom property values using log4net with this code:
LogicalThreadContext.Stacks["UserName"].Push("jsmith")

Elasticsearch receives this as

"UserName": {
  "Count": 1
},

expected is

"UserName": "jsmith",

For the reference, correct value ('jsmith' in the example above) gets logged by RollingFileAppender, so it looks like something in log4stash causing this issue. We use latest (1.1.1) log4stash version with MS webapi project targeting .NET 4.6. Our appender config looks like this:

<appender name="ElasticSearchAppender" type="log4net.ElasticSearch.ElasticSearchAppender, log4stash">
    <layout type="log4net.Layout.PatternLayout,log4net">
        <param name="ConversionPattern" value="%utcdate{ABSOLUTE} %-5level %logger - %message%newline" />
    </layout>
    <filter type="log4net.Filter.LevelRangeFilter">
        <levelMin value="WARN" />
        <levelMax value="FATAL" />
    </filter>
    <Server>localhost</Server>
    <Port>9200</Port>
    <IndexName>log_%{+yyyy-MM-dd}</IndexName>
    <IndexType>LogEvent</IndexType>
    <Bulksize>2000</Bulksize>
    <BulkIdleTimeout>10000</BulkIdleTimeout>
    <IndexAsync>True</IndexAsync>
    <FixedFields>772</FixedFields>
    <SerializeObjects>false</SerializeObjects>
    <ElasticFilters>
        <kv>
            <SourceKey>Message</SourceKey>
            <ValueSplit>|</ValueSplit>
            <FieldSplit> ,</FieldSplit>
        </kv>
    </ElasticFilters>
</appender>

Documentation

Hello,
is there any documentation available?
I want to use logstash for centralizing the logs of all my application, but I have no idea how to connect the server through log4stash to my programs.
greetings

cant setup

Thanks for this library I'm really glad to have found it. I'm having a problem setting up though, I simply can't get any log results.

Have a default Elastic Search 1.4 running on localhost:9200 and Kibana up on localhost:5601 under the "logstash-*" index pattern and it's not returning any results when I log (I'm logging to another location with a 2nd appender to verify the logs).

Here's my log4stash configuration:

<log4net>
    <appender name="ElasticSearchAppender" type="log4net.ElasticSearch.ElasticSearchAppender, log4stash">
      <Server>localhost</Server>
      <Port>9200</Port>
      <IndexName>logstash-%{+yyyy-MM-dd}</IndexName>
      <IndexType>LogEvent</IndexType>
      <ElasticFilters>
        <!-- example of using filter with default parameters -->
        <kv />
      </ElasticFilters>
    </appender>
    <appender name="Console" type="log4net.Appender.ConsoleAppender">
      <layout type="log4net.Layout.PatternLayout">
        <conversionPattern value="[%d][%-5p] %m%n" />
      </layout>
    </appender>
  </log4net>

ElasticSearchAppender does not handle property with null value

If I set the appender to submit the properties fields to elastic search and one of the properties has a null value then the appender fails to submit any data. In my case the builtin log4net property "log4net:UserName" had a null value, so it's not as simple as making sure I don't configure any properties with a null value since the value is coming from log4net.

It looks like this may be the issue:

if (FixedFields.ContainsFlag(FixFlags.Properties))
{
    var properties = loggingEvent.GetProperties();
    foreach (var propertyKey in properties.GetKeys())
    {
        logEvent[propertyKey] = properties[propertyKey].ToString();
    }
}

Specifically the call to properties[propertyKey].ToString() without checking that the value is not null first.

Thanks for all your work on this, it's a very useful library.

How do you pass in a Referrer Header?

How do you pass in a Referrer Header to ElasticSearch using Log4Stash?