sohlich/elogrus

Logrus Hook for ElasticSearch

elogrus's Issues

Apply for the maintainer

Hi Sohlich,
Nice meeting you. I leave you a message on the twitter, but I am not sure you will see the message, because you did not tweet much. So I create a issue here hoping you can contact me, I am using the elogrus, and I want to be maintainer for the project.

Liuzoxan

Unable to log to self defined index

I am able to successfully log data into the Elasticsearch if I do not define an index and let types be dynamically generated when using elogrus.

However with certain fields I would require different types, I have tried creating the index with my own mappings and even the mapping that resulted from being dynamically generated and in both instances elogrus no longer inserted new entries into the index.

How should I go about configuring mappings so that they are compatible with elogrus?

Error message when index generated has identical mapping to defined index:
Failed to fire hook: elastic: Error 400 (Bad Request): Rejecting mapping update to [log] as the final mapping would have more than 1 type: [_doc, log] [type=illegal_argument_exception]

go get error from gopkg.in

When I run go get gopkg.in/sohlich/elogrus,
there is an error.
package gopkg.in/sohlich/elogrus: unrecognized import path "gopkg.in/sohlich/elogrus" (parse https://gopkg.in/sohlich/elogrus?go-get=1: no go-import meta tags ()) .

Where is wrong?

Hostname in log is getting saved with the credentials

The Hostname info in the logs are getting added with the basic auth credential info .Is there any setting which can add only hostname and hide the credentials getting added to the index?

Push new tag for the current code

Gopkg.in is still serving an old version where the now unavailable github.com/Sirupsen/logrus is being used with the capital S.

I think a minor version upgrade should not break anything.

Elasticsearch 7.0 support

elastic v7 released. elogrus is not ready yet?

How do I modify index dynamically?

logrus's SetReportCaller invalid in elogrus

set log.SetReportCaller(true) can displayed log with caller
but elk has no caller infos

I'd like to add a mechanism for keeping track of the async hooks

Upon me trying to implement this project for myself with my companies elasticsearch, I ran into the issue of not being able to simply use the log.WHATEVER statement with a NewAsyncElasticHook. There was no built in method of keeping track of the number of messages still waiting to get sent, etc. so my project would always finish with logs not being sent.

So I'm fairly new to Go, but I think the way I'd like to tackle this is to implement a background channel that would be created upon the NewAsyncElasticHook that would send the logs that were gathered every 0.5 or whatever and the user could wait with a time limit on the channel to process all the logs and exit out if the time limit was reached.

What do you think? Worth implementing?

Fix timestamp to match default Kibana pattern

In Kibana, @timestamp is the default Time-field name
used for the time filter. Change the JSON output
to match ("@timestamp").

I'm put up the following Pull Request to address this:

#5

Does not create a new index for each new day.

At the moment all logs are stored as the same index.

Be better to create a new index for each day.

Report Caller missing

When I configure logrus in the following manner:

logrus.SetFormatter(&logrus.JSONFormatter{})
logrus.SetReportCaller(true)
logrus.SetOutput(os.Stdout)

And add the elogrus hook:

hook, err := elogrus.NewAsyncElasticHook(client, "localhost", logrus.TraceLevel, index)
logrus.AddHook(hook)

The report caller output is shown in Stdout, however it is not present in Elasticsearch/Kibana.

How can these fields (file and func) be included?

Elasticsearch 5.0

The go library for Elasticsearch is available for version 5, while elogrus still uses v3.

Are there any plans to support v5?

remove github.com module path from pkg.go.dev

Hi, pkg.go.dev maintainer here. We are serving pages for github.com/sohlich/elogrus, which isn't the right path for this module, but someone used it before you added a go.mod file so we can't notice that.

We would like to manually exclude this path from pkg.go.dev (while still keeping the correct gopkg.in path, of course). Do we have the owner's/maintainer's permission to do that?

New Trace log level in logrus

Hello, I'm just opening this issue to let you know, the next release of logrus (v1.2.0) will add a new trace level named Trace below Debug.
Here is the PR for reference sirupsen/logrus#844
You may want to take into account this new level.

Mapping types deprecated in Elastic v8

Version Info:
go 1.14.4 darwin/amd64

github.com/olivere/elastic/v7 v7.0.17
github.com/sirupsen/logrus v1.6.0
gopkg.in/sohlich/elogrus.v7 v7.0.0

After initializing a new client with elastic.SetInfoLog() writing to stdout, I got a warning:

Deprecation warning: 299 Elasticsearch-7.8.0-757314695644ea9a1dc2fecd26d1a43856725e65 "[types removal] Specifying types in document index requests is deprecated, use the typeless endpoints instead (/{index}/_doc/{id}, /{index}/_doc, or /{index}/_create/{id})."

Here's the announcement from Elastic and a related issue: olivere/elastic#1295

It's not a currently breaking things, but it's always nice to get ahead of these things, right?

My first thought was to just delete .Type("log") here:

elogrus/hook.go

Lines 215 to 222 in 7aa9ea8

	return func(entry *logrus.Entry, hook *ElasticHook) error {
	r := elastic.NewBulkIndexRequest().
	Index(hook.index()).
	Type("log").
	Doc(*createMessage(entry, hook))
	processor.Add(r)
	return nil
	}, err

I'm not sure how to prevent a breaking change, though. If someone is relying on the "log" type, could they just append the type to their index? Ex: {index}/{type}.

Properly handle error formatting

Heya,

Love the es hook, but errors are totally unusable, not sure if I configured something wrong.

 "error": {
        "error": {},
        "stack": [
          7960421,
          7982785,
          9199480,
          7986632,
          8154140,
          6701394,
          6686002,
          4539649
        ]
      },

Is the error I get in elasticsearch, not sure if I missconfigured something ? I'm also using github.com/pkg/errors and not just errors

missing Tracelevel from v7.0.0 version

There is an inconsistency between these two files.

https://github.com/sohlich/elogrus/blob/v7/hook.go
https://github.com/sohlich/elogrus/blob/v7.0.0/hook.go

In function newHookFuncAndFireFunc(line:108), logrus.Tracelevel is missing from the list and therefore Trace logs are not getting pushed to elastic.

Can i set username and password for my elasticsearch hosting ?

This is panic log "health check timeout: no Elasticsearch node available" from code
client, err := elastic.NewClient(elastic.SetURL(elasticHost), elastic.SetSniff(false))
I dont know how can i set Username and Password.

Name of Package logorus

e package name is: github.com/Sirupsen/logrus and the path of the logrus lib is github.com/sirupsen/logrus when I build I get an error

Fields are not compatible with ECS

ECS requires lowercase fields
https://www.elastic.co/guide/en/ecs/current/ecs-guidelines.html

The Message is uppercase and is not recognized by logs watcher

Build Fail in README example code

Hi, I try to build your README.md example code and got the following error,

./main.go:15:37: cannot use client (type *"gopkg.in/olivere/elastic.v5".Client) as type *"github.com/olivere/elastic".Client in argument to elogrus.NewElasticHook

How can I fix it?

Elastic 6.0 support

Is there any work going on for elastic 6.0

https://github.com/olivere/elastic

Hook not Fire()ed in a goroutine, causing application hangs

Hi,

As per the comment in the logrus hook code, it's expected that hooks will handle their own goroutines. It doesn't seem like you're firing your hook in a goroutine (and I can't see the elastic library doing so), so if ElasticSearch is running slowly for whatever reason, it will block until the requests are complete.

I'm not entirely sure if it's as simple as just setting the client...Do() call to run in a goroutine or not?

Thanks

no active connection found: no Elasticsearch node available

I'm using ELK (8) but the library returns.

no active connection found: no Elasticsearch node available

When browsing to the elasticsearch endpoint it does return a valid json response.

{
name: "elasticsearch-master-1",
cluster_name: "elasticsearch",
cluster_uuid: "ZqDoaZLyTe6O-rB6HQbscA",
version: {
number: "8.0.0-SNAPSHOT",
build_flavor: "default",
build_type: "docker",
build_hash: "747198ffed9b10273463367cf6ccc7ac5bfa077e",
build_date: "2021-02-12T18:02:06.143916Z",
build_snapshot: true,
lucene_version: "8.8.0",
minimum_wire_compatibility_version: "7.12.0",
minimum_index_compatibility_version: "7.0.0"
},
tagline: "You Know, for Search"
}