How does output to loki work?

[tsaikd]

How does output to loki work?

[helgeolav]

Hi

I had a quick look at this issue and from what I understand the loki output works and is passing data to the server? Everything except the geoenriched data.

The loki output does not support nested JSON structures, and "geoip" is not a simple type that can be processed. You have to copy all the fields like you do with geo2ip_city to get this right. And in the end you should delete the field "geoip" to get rid of the warning.

Disclaimer: I don't know anything about Loki but looked through the API documentation and I assume that for you to use this data it has to be passed in the stream attribute, which is done for all fields except the message field. Perhaps @yuangu can tell if loki supports nested inputs and if the module can be rewritten?

[KVInventoR]

hm, I also added:

  - remove_field:
    fields:
      - geoip

and in logs I got next:

2022/05/12 08:28:29 filtergeoip2.go:86 [info] geoip2 fsnotify initialized for /GeoLite2-City.mmdb
panic: interface conversion: interface {} is nil, not string

goroutine 1 [running]:
github.com/tsaikd/gogstash/config.GetFilters({0x13ca150, 0xc000380040}, {0xc0000a6180, 0x3, 0x13c6640?}, {0x13cb298, 0xc000312000})
        /build/config/filter.go:88 +0x42c
github.com/tsaikd/gogstash/config.(*Config).getFilters(...)
        /build/config/filter.go:104
github.com/tsaikd/gogstash/config.(*Config).startFilters(0xc000312000)
        /build/config/filter.go:108 +0x49
github.com/tsaikd/gogstash/config.(*Config).Start(0xc000312000, {0x13ca188?, 0xc000036040?})
        /build/config/config.go:138 +0x149
github.com/tsaikd/gogstash/cmd.gogstash({0x13ca188, 0xc000036040}, {0xc00003401f, 0xa}, 0xe0?, {0x0, 0x0}, 0x0)
        /build/cmd/gogstash.go:49 +0x285
github.com/tsaikd/gogstash/cmd.init.0.func2({0x13ca188, 0xc000036040}, 0x0?, {0x0?, 0x0?, 0xecb560?})
        /build/cmd/module.go:62 +0x85
github.com/tsaikd/KDGoLib/cliutil/cobrather.(*Module).MustNewCommand.func1(0xc00030c000?, {0x1c030e8, 0x0, 0x0})
        /go/pkg/mod/github.com/tsaikd/[email protected]/cliutil/cobrather/module.go:58 +0xa5
github.com/spf13/cobra.(*Command).execute(0xc00030c000, {0xc000030290, 0x0, 0x0})
        /go/pkg/mod/github.com/spf13/[email protected]/command.go:842 +0x67c
github.com/spf13/cobra.(*Command).ExecuteC(0xc00030c000)
        /go/pkg/mod/github.com/spf13/[email protected]/command.go:950 +0x39c
github.com/spf13/cobra.(*Command).Execute(...)
        /go/pkg/mod/github.com/spf13/[email protected]/command.go:887
github.com/tsaikd/KDGoLib/cliutil/cobrather.(*Module).MustMainRun(0x0?, {0x0?, 0x1b56628?, 0xc0000021a0?})
        /go/pkg/mod/github.com/tsaikd/[email protected]/cliutil/cobrather/module.go:130 +0x7e
main.main()
        /build/main.go:6 +0x27
2022/05/12 08:28:30 filtergeoip2.go:86 [info] geoip2 fsnotify initialized for /GeoLite2-City.mmdb
panic: interface conversion: interface {} is nil, not string

seems it's not possible to remove geoip map in simple way

[helgeolav]

This panic is part of initialization and should not occur. My first guess is a typo or indentation error in the config file. Could you provide the entire configuration file? Since geoip2 was initialized the error is below that section.

[yuangu]

I refer to this document that from https://grafana.com/docs/loki/latest/api/#post-lokiapiv1push 。In fact, it is found that the data received by Loki needs to be of type string. This is what ToStringE function does

[yuangu]

I think about how to solve this problem.

@helgeolav I looked at it because "geoip" is a nested type. However, Loki does not support nested types. Because it needs an accurate string value as the search key. I want to deal with it specially, but I'm afraid there will be some problems in the future.

How to solve it?

for key, value := range event.Extra {
		if key == "geoip" {
			for gp_key, gp_value := range value.(map[string]interface{}) {
				v, err := ToStringE(gp_value)
				if err != nil {
					goglog.Logger.Warnf("geoip_key: %v error:%v", gp_key, err)
				}
				_stream[key+"_"+gp_key] = v
			}
			continue
		}

		v, err := ToStringE(value)
		if err != nil {
			goglog.Logger.Warnf("key: %v error:%v", key, err)
		}
		_stream[key] = v
	}

[KVInventoR]

@helgeolav this is my current config:

 cat config/config.yml
---
#

chsize: 1000

input:
  - type: file
    path: /logging/test_input.log
    start_position: end
    sincedb_write_interval: 5

filter:
  - type: grok
    match: ["%{IPTABLES_SRC}"]
    source: "message"
    patterns_path: "/iptables.grok"

  - type: geoip2
    db_path: "/GeoLite2-City.mmdb"
    ip_field: "src_ip"
    cache_size: 100000
    key: geoip
    flat_format: true

  - remove_field:
    fields:
      - geoip

output:
  - type: stdout
    codec: json

yamllint config/config.yml && echo $?
0

there are no yaml errors

[KVInventoR]

oh, sorry, I found an issue with remove_field

  - type: remove_field
    fields:
      - geoip

[helgeolav]

oh, sorry, I found an issue with remove_field

  - type: remove_field
    fields:
      - geoip

I see it now. That kind of configuration errors are easy to overlook. We should look into the code and rewrite it so the error is more clear than just a panic.

Does it work better for you now?

[KVInventoR]

Hi, please excuse the long-time delay, I switched to another project.
Currently, going to restore work on this solution for monitoring system.