faceair / victorialogs Goto Github PK
View Code? Open in Web Editor NEWLoki On VictoriaMetrics
License: Apache License 2.0
Loki On VictoriaMetrics
License: Apache License 2.0
Hello @faceair - Great project here! I run cLoki, similiar initiative in NodeJS for clickhouse.
Could you share or add the intended basic build instructions to generate and test a local binary?
I've been playing with this and querying seems fundamentally broken. I've built from db39e42 and have been sending data with netcat
. I get a strange issue where characters from log lines overwrite characters in other log lines.
Here's a quick and nasty script to demonstrate:
pkill vmstorage
pkill vmselect
pkill vminsert
sleep 5
rm -rf vmstorage-data/
nohup bin/vmstorage > vmstorage.log 2>&1 &
nohup bin/vmselect -storageNode 127.0.0.1:8401 > vmselect.log 2>&1 &
nohup bin/vminsert -storageNode 127.0.0.1:8400 -importerListenAddr 127.0.0.1:2003 > vminsert.log 2>&1 &
sleep 30
echo 'loki{foo="bar"} "The course of true love never did run smooth."' | nc -vN 127.0.0.1 2003
sleep 1
curl -s http://127.0.0.1:8481/select/0/loki/api/v1/query_range --data-urlencode 'query={foo="bar"}' --data-urlencode 'step=60' | jq
echo 'loki{foo="bar"} "All that glisters is not gold."' | nc -vN 127.0.0.1 2003
sleep 2
curl -s http://127.0.0.1:8481/select/0/loki/api/v1/query_range --data-urlencode 'query={foo="bar"}' --data-urlencode 'step=60' | jq
echo 'loki{foo="bar"} "Whats in a name? That which we call a rose By any other name would smell as sweet."' | nc -vN 127.0.0.1 2003
sleep 3
curl -s http://127.0.0.1:8481/select/0/loki/api/v1/query_range --data-urlencode 'query={foo="bar"}' --data-urlencode 'step=60' | jq
echo 'loki{foo="bar"} "Once more unto the breach, dear friends, once more, Or close the wall up with our English dead!"' | nc -vN 127.0.0.1 2003
sleep 4
curl -s http://127.0.0.1:8481/select/0/loki/api/v1/query_range --data-urlencode 'query={foo="bar"}' --data-urlencode 'step=60' | jq
echo 'loki{foo="bar"} "By the pricking of my thumbs, Something wicked this way comes. Double, double toil and trouble; Fire burn, and cauldron bubble."' | nc -vN 127.0.0.1 2003
sleep 5
curl -s http://127.0.0.1:8481/select/0/loki/api/v1/query_range --data-urlencode 'query={foo="bar"}' --data-urlencode 'step=60' | jq
echo 'loki{foo="bar"} "log log log log"' | nc -vN 127.0.0.1 2003
curl -s http://127.0.0.1:8481/select/0/loki/api/v1/query_range --data-urlencode 'query={foo="bar"}' --data-urlencode 'step=60' | jq
sleep 15
curl -s http://127.0.0.1:8481/select/0/loki/api/v1/query_range --data-urlencode 'query={foo="bar"}' --data-urlencode 'step=60' | jq
The expected final response is 5 Shakespeare quotes and the line "log log log log" but what we actually get is:
{
"status": "success",
"data": {
"resultType": "streams",
"result": [
{
"stream": {
"__name__": "loki",
"foo": "bar"
},
"values": [
[
"1606968840550000000",
"By the pricking of my thumbs, Something wicke"
],
[
"1606968841583000000",
" this way comes. Double, doubl"
],
[
"1606968843619000000",
" toil and trouble; Fire burn, and cauldron bubble.\u000flog log log log smell as sweet."
],
[
"1606968846650000000",
"Once more unto the breach, dear friends, once more, Or close the wall up with our English dead!"
],
[
"1606968850686000000",
"By the pricking of my thumbs, Something wicked this way comes. Double, double toil and trouble; Fire burn, and cauldron bubble."
],
[
"1606968855719000000",
"log log log log"
]
]
}
]
}
}
You'll notice that my "log" messages are truncated, repeating, missing and (most surprising of all, "log log log log"
is overlaid over the top of (half) a Shakespeare quote: "toil and trouble; Fire burn, and cauldron bubble.\u000flog log log log smell as sweet."
If you review the output of vminsert
(e.g with tail vminsert.log
), there was no problems reported writing the logs (e.g I've seen missing a end quote char will correctly return an error about being unable to umarshal.)
In Grafana, it looks like this (now-15m to now
):
Furthermore, if I change the Time Range in Grafana, entire log lines disappear:
now-1h to now
:
Note: There are less lines than above!
Are there plans to update the VM code in this repository to support latest code?
I just cloned master
and built with make all
and got the following error:
$ bin/vmselect -storageNode 127.0.0.1:8401
panic: BUG: metric "vm_cache_requests_total{type=\"promql/regexp\"}" is already registered
goroutine 1 [running]:
github.com/VictoriaMetrics/metrics.(*Set).mustRegisterLocked(0xc000146300, 0xa3b576, 0x2d, 0xae3120, 0xc00012c070)
/root/go/pkg/mod/github.com/!victoria!metrics/[email protected]/set.go:450 +0x2ae
github.com/VictoriaMetrics/metrics.(*Set).registerMetric(0xc000146300, 0xa3b576, 0x2d, 0xae3120, 0xc00012c070)
/root/go/pkg/mod/github.com/!victoria!metrics/[email protected]/set.go:433 +0xda
github.com/VictoriaMetrics/metrics.(*Set).NewGauge(0xc000146300, 0xa3b576, 0x2d, 0xc000110ea0, 0x0)
/root/go/pkg/mod/github.com/!victoria!metrics/[email protected]/set.go:250 +0x89
github.com/VictoriaMetrics/metrics.NewGauge(...)
/root/go/pkg/mod/github.com/!victoria!metrics/[email protected]/gauge.go:24
github.com/VictoriaMetrics/VictoriaLogs/lib/logql.glob..func1(0x7ff8000000000001)
/usr/share/victorialogs/VictoriaLogs/lib/logql/regexp_cache.go:36 +0xbe
github.com/VictoriaMetrics/VictoriaLogs/lib/logql.init()
/usr/share/victorialogs/VictoriaLogs/lib/logql/regexp_cache.go:46 +0xa39
root@victorialogs-0:/usr/share/victorialogs/VictoriaLogs#
To fix it, I had to do the following in lib/logql/regexp_cache.go
:
- "github.com/VictoriaMetrics/metrics"
and
- metrics.NewGauge(`vm_cache_requests_total{type="promql/regexp"}`, func() float64 {
- return float64(rc.Requests())
- })
- metrics.NewGauge(`vm_cache_misses_total{type="promql/regexp"}`, func() float64 {
- return float64(rc.Misses())
- })
- metrics.NewGauge(`vm_cache_entries{type="promql/regexp"}`, func() float64 {
- return float64(rc.Len())
- })
After I did that, it make all
ran successfully and the binaries appeared in bin/
.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.