janianttonen/winston-loki

Grafana Loki transport for the nodejs logging library Winston.

winston-loki

A Grafana Loki transport for the nodejs logging library Winston.

Usage

This Winston transport is used similarly to other Winston transports. Require winston and define a new LokiTransport() inside its options when creating it.

Examples

Several usage examples with a test configuration for Grafana+Loki+Promtail reside under examples/. If you want the simplest possible configuration, that's probably the place to check out. By defining json: true and giving winston-loki the correct host address for Loki is enough for most.

Options

LokiTransport() takes a Javascript object as an input. These are the options that are available, required in bold:

Parameter	Description	Example	Default
host	URL for Grafana Loki	http://127.0.0.1:3100	null
interval	The interval at which batched logs are sent in seconds	30	5
json	Use JSON instead of Protobuf for transport	true	false
batching	If batching is not used, the logs are sent as they come	true	true
clearOnError	Discard any logs that result in an error during transport	true	false
replaceTimestamp	Replace any log timestamps with Date.now(). Warning: Disabling replaceTimestamp may result in logs failing to upload due to recent changes in the upstream Loki project. It is recommended to leave this option enabled unless you have a specific reason to disable it.	true	true
labels	custom labels, key-value pairs	{ module: 'http' }	undefined
format	winston format (https://github.com/winstonjs/winston#formats)	simple()	undefined
gracefulShutdown	Enable/disable graceful shutdown (wait for any unsent batches)	false	true
timeout	timeout for requests to grafana loki in ms	30000	undefined
basicAuth	basic authentication credentials to access Loki over HTTP	username:password	undefined
onConnectionError	Loki error connection handler	(err) => console.error(err)	undefined
useWinstonMetaAsLabels	Use Winston's "meta" (such as defaultMeta values) as Loki labels	true	false
ignoredMeta	When useWinstonMetaAsLabels is enabled, a list of meta values to ignore	["error_description"]	undefined

Example

With default formatting:

const { createLogger, transports } = require("winston");
const LokiTransport = require("winston-loki");
const options = {
  ...,
  transports: [
    new LokiTransport({
      host: "http://127.0.0.1:3100"
    })
  ]
  ...
};
const logger = createLogger(options);

You can set custom labels in every log as well like this:

logger.debug({ message: 'test', labels: { 'key': 'value' } })

TODO: Add custom formatting example

Developing

Requirements

Running a local Loki for testing is probably required, and the easiest way to do that is to follow this guide: https://github.com/grafana/loki/tree/master/production#run-locally-using-docker. After that, Grafana Loki instance is available at http://localhost:3100, with a Grafana instance running at http://localhost:3000. Username admin, password admin. Add the Loki source with the URL http://loki:3100, and the explorer should work.

Refer to https://grafana.com/docs/loki/latest/api/ for documentation about the available endpoints, data formats etc.

Example

npm install
npm link
cd ~/your_project
npm link winston-loki
npm install

And you should have a working, requirable winston-loki package under your project's node_modules. After the link has been established, any changes to winston-loki should show on rerun of the software that uses it.

Run tests

npm test

Write new ones under /test

winston-loki's Issues

Add better documentation

Comment code and define parameters

TypeScript typings

I am writing TypeScript typings for winston-loki right now. Would you like me to open a PR for them 🤔 ?

useCustomFormat is broken

As it has been found during the discussion in PR there is a problem with the way how Loki message is getting constructied from the winston's info object:

const line = this.useCustomFormat ? info[MESSAGE] : <some custom serialization of info>

Where useCustomFormat is true when some format has been provided to the winston-loki transport instance and false otherwise.

Notes that winston has a concept of "finalizing" and "non-finalizing" formats: if a format is finalizing - it's updating the value of info[MESSAGE]. If a format is non-finalizing - it's not updating info[MESSAGE]. info[MESSAGE] has some default value even without applying any formats.

There are two problematic scenarios:

1. Only non-finalizing format(s) has been used on the transport level.

It means that useCustomFormat is true and info[MESSAGE] has default value. This default value doesn't include metadata like level and any changes being done by all non-finalizing formats from both logger and transport levels. And this default value will be sent to Loki.

2. A finalizing format has been used on the logger level and no format has been used on the transport level.

It means that useCustomFormat is false but info[MESSAGE] has some prepared value. This value will be lost during the serialization of info.

Of course, the correct behavior would be to use info[MESSAGE] if a finalizing format has been applied and try to serialize the info object somehow if there were no finalizing format applied. But there is no way to understand if it has been used or not.

It seems that the only way to avoid unexpected behavior is to remove useCustomFormat, always use info[MESSAGE], and explicitly tell in the documentation that some finalizing format must be used.

In most cases, winston.format.json() is exactly what most users would expect to see - we can recommend using it unless there is an explicit need to use some other finalizing format.

After the discussion, I'm willing to make a PR with a fix.

If options.interval is not specified and sendBatchToLoki throws, interval becomes 0

this.options.interval may be undefined:

winston-loki/src/batcher.js

Lines 40 to 43 in fda379c

	this.interval = this.options.interval
	? Number(this.options.interval) * 1000
	: 5000
	this.circuitBreakerInterval = 60000

this.interval is overwritten if await this.sendBatchToLoki() throws:

winston-loki/src/batcher.js

Lines 228 to 230 in fda379c

	} catch (e) {
	this.interval = this.circuitBreakerInterval
	}

Number(undefined) becomes 0

winston-loki/src/batcher.js

Line 226 in fda379c

this.interval = Number(this.options.interval) * 1000

Endless batching until out of memory if request to grafana loki won't hang up

If the request to grafana loki get's stuck for whatever reason, winston-loki will batch the logs until we run out of memory, since we don't have any timeout on our request defined.

As a first step I created a PR to make the timeout configurable: #55

We might want to discuss if there is a sensible timeout we want to set as a default value.

Checklist

• Modify src/requests.js ! No changes made Edit
• Running GitHub Actions for src/requests.js ✗ Edit
• Modify src/batcher.js ! No changes made Edit
• Running GitHub Actions for src/batcher.js ✗ Edit
• Modify index.js ! No changes made Edit
• Running GitHub Actions for index.js ✗ Edit
• Modify index.d.ts ✓ e9bd74a Edit
• Running GitHub Actions for index.d.ts ✓ Edit
• Modify README.md ! No changes made Edit
• Running GitHub Actions for README.md ✗ Edit

Should this work with Grafana Cloud? I'm not getting logs

I'm trying to use this library to publish logs to Grafana Cloud, this is my config:

...
const lokiTransport = new LokiTransport({
  level: 'info',
  batching: false,
  host: "https://logs-prod-us-central1.grafana.net",
  json: true,
  basicAuth: '<my_user>:<my_key>',
});
logger.add(lokiTransport);

but I'm not getting any logs in grafana.

Package size optimizations / futureproofing

• Remove got dependency, move to using phin or just plain node http(s) requests
• Remove moment dependency, move to date-fns

basic authentication header fails due to extra colon

I've tried using the basic authentication feature and failed.
It seems to be due to this line

When removing the colon following basic the log is sent successfully.
I'll open a PR now to fix the issue.

Publish debug informations through the `debug` package

Hi! 👋

Firstly, thanks for creating this life-saving package! 🙂

Today I used patch-package to patch [email protected] for the project I'm working on.

My problem was that I added a double-slash at the end of my Loki URL.
e.g. I supplied http://host/ instead of http://host

The patch I'm submitting will help others debug this kind of problem in the future.
I suggest using the almost-standard debug package to supply the user with debugging information.

To enable printing of the debugging information, the user only has to define the following environment variable:

DEBUG=winston-loki

By using this environment variable, the user will get debug information printed to the console like so:

Here is a rough implementation example:

diff --git a/node_modules/winston-loki/src/requests.js b/node_modules/winston-loki/src/requests.js
index 5f57fdc..18a7af3 100644
--- a/node_modules/winston-loki/src/requests.js
+++ b/node_modules/winston-loki/src/requests.js
@@ -1,5 +1,6 @@
 const http = require('http')
 const https = require('https')
+const debug = require('debug')('winston-loki')
 
 const post = async (lokiUrl, contentType, headers = {}, data = '', timeout) => {
   // Construct a buffer from the data string to have deterministic data size
@@ -25,6 +26,8 @@ const post = async (lokiUrl, contentType, headers = {}, data = '', timeout) => {
       timeout: timeout
     }
 
+    debug("sending options: " + JSON.stringify(options, undefined, 2))
+
     // Construct the request
     const req = lib.request(options, res => {
       let resData = ''
@@ -34,6 +37,7 @@ const post = async (lokiUrl, contentType, headers = {}, data = '', timeout) => {
 
     // Error listener
     req.on('error', error => {
+      debug("loki HTTP error: " + error)
       reject(error)
     })
 

I'm interested in creating a PR for this feature.

Thank you for your attention.

Issue when implementing graceful shutdown

I'm implementing graceful shutdown of my app, but found an issue might be related to this project.

Minimal Reproducible Code

"use strict";
require("dotenv").config();

const winston = require("winston");
const { combine, timestamp, label, printf, colorize } = winston.format;
const LokiTransport = require("winston-loki");
const env = process.env.NODE_ENV || "development";
const level = env === "development" ? "debug" : "info";
const name = "express";

const Log = winston.createLogger({
    level : level,
    transports: [
        new winston.transports.Console({
            format: combine(
                label({ label: name }),
                colorize(),
                timestamp(),
                printf(({ level, message, label, timestamp }) => {
                    return `${timestamp} ${level}: [${label}] ${typeof message == "string" ? message : JSON.stringify(message)}`;
                }),
            ),        
        }),
        // works if commented
        new LokiTransport({
            host: process.env.GRAFANA_LOKI_URL
        }),
    ]
});
const express = require("express");
const app = express();
const server = require("http").Server(app);

server.listen(8080, () => {
    Log.info("The server is running...");
});

const gracefullyShutdown = () => {
    Log.info(`SIGTERM / SIGINT signal received.`);
    Log.info('Closing http server.');
    server.close(() => {
        Log.info('Http server closed.');

        setTimeout(()=> {
            Log.info('Shutdown after 5 secs!');
            
        }, 5000);
    });
};

process.on('SIGTERM', gracefullyShutdown);
process.on('SIGINT' , gracefullyShutdown);

process.once('exit', () => {
    console.trace("I am here");
})

process.on('unhandledRejection', (reason, p) => {
    console.log('Unhandled Rejection at:', p, 'reason:', reason);
});

Explanation

1. if I add LokiTransport to winston transport as my code shown, when I press Ctrl + C to trigger SIGINT, the app exit immediately without waiting. here is the log:

2020-02-12T13:20:25.440Z info: [express] The server is running...
^C2020-02-12T13:20:27.553Z info: [express] SIGTERM / SIGINT signal received.
2020-02-12T13:20:27.553Z info: [express] Closing http server.
2020-02-12T13:20:27.553Z info: [express] Http server closed.
Trace: I am here
    at process.<anonymous> (/Users/user/abcproject/index.js:54:13)
    at Object.onceWrapper (events.js:300:26)
    at process.emit (events.js:215:7)
    at exit (internal/process/per_thread.js:158:15)
    at processTicksAndRejections (internal/process/task_queues.js:75:11)

2. If I commented LokiTransport, when I press Ctrl + C to trigger SIGINT, the app exit after 5s as my expected. here is the log:

2020-02-12T13:18:51.733Z info: [express] The server is running...
^C2020-02-12T13:18:53.706Z info: [express] SIGTERM / SIGINT signal received.
2020-02-12T13:18:53.707Z info: [express] Closing http server.
2020-02-12T13:18:53.707Z info: [express] Http server closed.
2020-02-12T13:18:58.713Z info: [express] Shutdown after 5 secs!
Trace: I am here
    at process.<anonymous> (/Users/user/abcproject/index.js:68:13)
    at Object.onceWrapper (events.js:300:26)
    at process.emit (events.js:210:5)

Is there any possible reason to explain this behavior? Maybe because of some dependency library?
Or any way to find out what code causes early exit?

Thanks

JSON transport fails silently

It seems the JSON request is failing silently.

reqBody={"streams":[{"labels":"{job=\"undefined\", level=\"info\"}","entries":[{"line":"Listening on port: 8000 "}]}]}

The got request is rejected with this error:

{ HTTPError: Response code 500 (Internal Server Error)
    at EventEmitter.emitter.on (/home/dev/node_modules/got/source/as-promise.js:74:19)
    at <anonymous>
    at process._tickCallback (internal/process/next_tick.js:189:7)
  name: 'HTTPError',
  host: 'localhost:3100',
  hostname: 'localhost',
  method: 'POST',
  path: '/api/prom/push',
  socketPath: undefined,
  protocol: 'http:',
  url: 'http://localhost:3100/api/prom/push',
  gotOptions: 
   { path: '/api/prom/push',
     protocol: 'http:',
     slashes: true,
     auth: null,
     host: 'localhost:3100',
     port: '3100',
     hostname: 'localhost',
     hash: null,
     search: null,
     query: null,
     pathname: '/api/prom/push',
     href: 'http://localhost:3100/api/prom/push',
     retry: 
      { retries: [Function],
        methods: [Object],
        statusCodes: [Object],
        errorCodes: [Object] },
     headers: 
      { 'user-agent': 'got/9.6.0 (https://github.com/sindresorhus/got)',
        'content-type': 'application/json',
        'accept-encoding': 'gzip, deflate',
        'content-length': 110 },
     hooks: 
      { beforeRequest: [],
        beforeRedirect: [],
        beforeRetry: [],
        afterResponse: [],
        beforeError: [],
        init: [] },
     decompress: true,
     throwHttpErrors: true,
     followRedirect: true,
     stream: false,
     form: false,
     json: false,
     cache: false,
     useElectronNet: false,
     body: '{"streams":[{"labels":"{job=\\"undefined\\", level=\\"info\\"}","entries":[{"line":"Listening on port: 8000 "}]}]}',
     method: 'POST' },
  statusCode: 500,
  statusMessage: 'Internal Server Error',
  headers: 
   { 'content-type': 'text/plain; charset=utf-8',
     'x-content-type-options': 'nosniff',
     date: 'Tue, 29 Jan 2019 08:47:47 GMT',
     'content-length': '52',
     connection: 'close' },
  body: 'rpc error: code = Unknown desc = Entry out of order\n' }

The same log entries with the protobuf gets stored correctly in Loki.

This may be an issue on Loki's API side. Feel free to close if this is true.

JSON mode & Batching is not working

winston-loki/src/batcher.js

Lines 144 to 155 in f291979

	let reqBody
	// If the data format is JSON, there's no need to construct a buffer
	if (this.options.json) {
	if (logEntry !== undefined) {
	// If a single logEntry is given, wrap it according to the batch format
	reqBody = JSON.stringify({ streams: [logEntry] })
	} else {
	// Stringify the JSON ready for transport
	reqBody = JSON.stringify(reqBody)
	}
	} else {

I think line 153, should be reqBody = JSON.stringify(this.batch), or the entire reqBody would be undefined.

Confusing options batching & gracefulShutdown

winston-loki/index.js

Lines 21 to 30 in 024f508

	this.batcher = new Batcher({
	host: options.host,
	interval: options.interval,
	json: options.json,
	batching: !options.batching,
	clearOnError: options.clearOnError,
	replaceOnError: options.replaceOnError,
	replaceTimestamp: options.replaceTimestamp,
	gracefulShutdown: !options.gracefulShutdown
	})

You use ! in front of batching & gracefulShutdown, which means it's inverted.
So should I put true or false?

the results are:
In / out
undefined -> true
false->true
true->false

QUESTION: How many log levels are supported?

When using the logger I am only able to get a response from 3 log level functions:

logger.info()
logger.warn()
logger.error()

Are there any other log levels that im missing?

Can it used with frontend like react ?

Can this wrapper used with the front end framework or library like react or angular ?

Race condition in batching leading to lost logs

In batching mode, clearBatch is being called after the post request:

winston-loki/src/batcher.js

Line 198 in 67e5fdf

logEntry === undefined && this.clearBatch()

Logs that came in while the post request is pending will be lost. clearBatch needs to be called before.

data is not being send to promtail

I used the sample in README and there is no error or something but data is not being send to promtail.
I also checked with http://localhost:3100/loki/api/v1/labels

Can not find any logs in Grafana Explorer

I run Grafana Loki in my host machine using Docker, and I have a worker program also running in Docker.

CONTAINER ID   IMAGE                        COMMAND                  CREATED       STATUS       PORTS                                                           NAMES
fe033b5de44b   grafana/grafana-enterprise   "/run.sh"                2 hours ago   Up 2 hours   0.0.0.0:3333->3000/tcp, :::3333->3000/tcp                       grafana
f023d89739e1   grafana/promtail:2.3.0       "/usr/bin/promtail -…"   6 days ago    Up 6 days                                                                    condescending_goodall
4e1a1d5122c7   grafana/loki:2.3.0           "/usr/bin/loki -conf…"   6 days ago    Up 6 days    0.0.0.0:3100->3100/tcp, :::3100->3100/tcp                       adoring_johnson

I set up Winston Loki use follow configs:

new LokiTransport({
  level: 'verbose',
  json: true,
  labels: { job: 'MetaCMSWorkerGit' },
  host: 'http://172.17.0.1:3100',
  handleExceptions: true,
  handleRejections: true,
}),

In worker docker stdout, I can saw LokiTransport options, batcher and pushLogEntry has correct value (I add some console log into winston-loki package):

LokiTransport options: {
  level: 'verbose',
  json: true,
  labels: { job: 'MetaCMSWorkerGit' },
  host: 'http://172.17.0.1:3100',
  handleExceptions: true,
  handleRejections: true
}
this.batcher: {
  level: 'verbose',
  json: true,
  labels: { job: 'MetaCMSWorkerGit' },
  host: 'http://172.17.0.1:3100',
  handleExceptions: true,
  handleRejections: true
}
winston:create-logger: Define prototype method for "error"
winston:create-logger: Define prototype method for "warn"
winston:create-logger: Define prototype method for "info"
winston:create-logger: Define prototype method for "http"
winston:create-logger: Define prototype method for "verbose"
winston:create-logger: Define prototype method for "debug"
winston:create-logger: Define prototype method for "silly"
pushLogEntry: {
  labels: { level: 'info', job: 'MetaCMSWorkerGit' },
  entries: [
    {
      ts: '08/30/2021, 10:59:31 AM',
      line: 'Log files saved to /tmp/metacmsworkergit-WH1oq6 {"host":"","runtime":{"pid":320463,"platform":"linux","versions":{"node":"14.17.3","v8":"8.4.371.23-node.67","uv":"1.41.0","zlib":"1.2.11","brotli":"1.0.9","ares":"1.17.1","modules":"83","nghttp2":"1.42.0","napi":"8","llhttp":"2.1.3","openssl":"1.1.1k","cldr":"39.0","icu":"69.1","tz":"2021a","unicode":"13.0"}},"context":"main","ms":"+0ms"}'
    }
  ]
}
winston:file: written true false

But I can not find any log in Grafana Explorer, did I missed something?

use Symbol.for('level') when extracting label

following line extract the level information from the info object and uses them as the level label

https://github.com/JaniAnttonen/winston-loki/blob/development/index.js#L58

this causes problems when using

• colorized text
• text that is padded for formation
• levels that are lower or uppercase

instead the Symbol.for('level') should be used, as per docu of winston::
https://github.com/winstonjs/winston#streams-objectmode-and-info-objects

Checklist

• Modify index.js ✓ e35241a Edit
• Running GitHub Actions for index.js ✓ Edit

Labels not correctly stringified in request

It's wierd. I found stream label in request body is "[object Object]".

{"streams":[{"stream":"[object Object]","values":[["1634641025327000000","{\"level\":\"info\",\"message\":\"haha\"}"]]}]}

My code:

new LokiTransport({
            host: 'http://192.168.11.204:3100',
            format: format.combine(
                format.json(),
            ),
            batching: false,
            labels: {
                biz: 'test',
            },
            json: true
        })

And it works when I remove the code below.

winston-loki/index.js

Line 73 in 0eb49ec

lokiLabels = JSON.parse(JSON.stringify(lokiLabels, (key, value) => value ? value.toString() : value))

Feature Request: awaitable finishRequests method

We have third party libraries that call process.exit which seems to be incompatible with the graceful shutdown option due to the nature of async-exit-hooks. I am currently achieving this with:

const lokiTranport = LokiTransport()
... // make some logger calls
async function finishLokiRequest() {
  return new Promise(async (resolve) => {
    await lokiTransport.batcher.sendBatchToLoki()
    resolve()
  })
}
await finishLokiRequest()

Hoping to get something like this made part of the public API

Cypress test failing due to snappy update (probably)

Hi, thanks for the new release. Unfortunately it broke our builds. I don't know what exactly is going wrong, but it seems to be related to snappy, and as the new 6.0.4 release includes a new snappy version that is my main suspect.
I run cypress tests of my project using cypress 6.5.0 and the @cypress/browserify-preprocessor
Since the release of 6.0.4 my cypress_console.log file lists:

Oops...we found an error preparing this test file:

  cypress/support/index.js

The error was:

SyntaxError: Unexpected character '�' (1:2) while parsing /e2e/node_modules/@napi-rs/snappy-win32-x64-msvc/snappy.win32-x64-msvc.node while parsing file: /e2e/node_modules/@napi-rs/snappy-win32-x64-msvc/snappy.win32-x64-msvc.node

This occurred while Cypress was compiling and bundling your test code. This is usually caused by:

- A missing file or dependency
- A syntax error in the file or one of its dependencies

Fix the error in your code and re-run your tests.

We are not using snappy, and are telling browserify to not complain about missing libraries. But I've got the feeling somehow it ended up still including some binary where it shouldn't.

I've already spent too much time trying to find the issue, so for now we will use my fork of this project instead. But I do hope that this bugreport helps finding and solving this problem.

I cannot start project

Hi,

I'm trying to get the project up and I don't know how to do it.

$ npm run
Lifecycle scripts included in [email protected]:
  test
    ./node_modules/.bin/standard && ./node_modules/.bin/jest

available via `npm run-script`:
  build:proto
    ./node_modules/.bin/pbjs -t static-module -w commonjs -o ./src/proto/index.js ./src/proto/logproto.proto && npm run lint:fix
  lint
    ./node_modules/.bin/eslint *
  lint:fix
    ./node_modules/.bin/standard --fix
  coverage
    jest --coverage
  coveralls
    jest --coverage && cat ./test/coverage/lcov.info | coveralls

winston-loki not working properly with Grafana v7.1.3

Hello, I have a working application using winston-loki, but after updating the grafana server to a new version (7.1.3) I found out that most part of the log messages are not being logged in the grafana server.

I've tried the winston-loki v5.1.2 (some messages are logged in grafana) and winston-loki v6.0.0-rc.4 (no messages are logged in grafana).

Any help? When u said that the latest winston-loki should work within the latest grafana server, which version of grafana are u talking about?

Thanks in advance!

How to configure the log properly

I think the documentation of this project is very poor. It does not give any example about how to configure and use the logger properly.

This is what I have configured for the logger:

const {createLogger, format, transports} = require('winston')
const LokiTransport = require("winston-loki")
const {combine, timestamp, json, prettyPrint} = format

const logger = createLogger({
  level: 'info',
  format: combine(
    timestamp(),
    json()
  ),
  transports: [
    new transports.Console({
      level: 'debug',
      format: combine(
        prettyPrint()
      )
    }),
    new LokiTransport({
      level: 'info',
      host: "http://localhost:3100",
    })
  ]
})

module.exports = logger

Importing the logger into another module and calling:

logger.info('Starting API...', {job: 'strenux'})

... I can see the logs in the console, but nothing appears in the Loki server, which is up and running at http://localhost:3100.

Loki response status codes 301 and 400 code do not lead to failure workflow

winston - 3.3.3
winston-loki - 6.0.3
transport - http

Hello! When Loki returns status code 301 or 400 for POST /loki/api/v1/push transport request does not emit error event and batcher treats this case as successful sending. Looks like this behavior is unexpected, can you please comment on that?

Update documentation & readme for 6.0.0

There might be some old information regarding the configuration.

Sending to Grafana Cloud Loki instance not working

I'm trying to send logs to the SaaS Loki instance of grafana cloud.

the url ist like this:
https://<user id>:<api key>@logs-prod-us-central1.grafana.net,

the api key look like follows (notice the = in the end):
eyJrIjoiYzhmZTNiZDJhNDc1YTg4MTU5ZmRjZjRkM2M4YzVjM2NjZjQ3NzcyNSIsIm4iOiJ0ZXN0IiwiaWQiOjUxOTQ5MH0=

in batcher.js on line 30 a url object is created from this, adding '/loki/api/v1/push'. It seems that it is url_encoded in this step because when read back from the object the url now is like:
https://77571:<...>TQ5MH0%[email protected]/

can this be fixed or is there a way around this issue?

best, fabian

is this repo still working?

as grafana stated in their official website their api can undergo breaking changes,
and they didn't even listed this project even as an unofficial related library to their project

i hoped i could use this lib (which is currently most popular node lib related to loki) to use it's functionality but i struggled for +8 hours and couldn't make it work,

it's not failing fast i can't even tell if it's facing any issue or not, tried different configurations then used the provided example and tests and yet i can't get what i need

Logs with labels that have a number as value fail silently

Heya

When using code like the following, logs do not make it to Loki. Other logs were fine, but this particular line was never ending up in Loki.

  logger.debug({
    message: 'bla bla bla',
    labels: { serverId: 2 }
  });

After some (read: a lot) of debugging I found that in the then here, the response is not used. However, adding a var there and debugging reveals an error message in the http response:

"loghttp.PushRequest.Streams: []*loghttp.Stream: loghttp.Stream.Entries: []loghttp.Entry: Labels: ReadString: expects \" or n, but found 2, error found in #10 byte of ...|erverId\":2,\"serverNa|..., bigger context ...|\"streams\":[{\"stream\":{\"level\":\"debug\",\"serverId\":2,\"serverName\":\"http://127.0.0.1:1337\",\"release\":\"1|...\n"

It seems to me that Loki returns a 200 on error, which causes the code not to fall into the catch block. I didn't verify this though :)

I fixed this on my side by making sure any labels I send are strings (and not numbers) but it'd be great if the library has a fallback for this. Or could propagate the error so the developer knows something is wrong.

Thanks for a great library 😃

document header options

It is possible to pass headers as options to the request. This should be documented in the Options part.

Flush queue on process exit

When using batching and the process exits between two flush operations, the most recent logs are lost (not being sent to Loki).

v6.0.2 Logs are accepted, but actually not stored in Loki

My setup:

docker run -d -p 3100:3100 grafana/loki

import winston from 'winston'
import Loki from 'winston-loki'

const logger = winston.createLogger({
  transports: [
    new Loki({
      host: 'http://127.0.0.1:3100',
    }),
  ],
})

logger.info("test")

The JSON payload sent to Loki in v6.0.2 contains "[Object object]" instead of the actual labels payload. This was introduced by exactly this change ec82e79

The payload sent to Loki is:

{
  "streams": [
    {
      "stream": "[object Object]",
      "values": [
        [
          "1634077254005000000",
          "test"
        ]
      ]
    }
  ]
}

There is no http error response, but these are impossible to find in grafana.

Winston-loki not working with Grafana Cloud Loki

If you're running Grafana & Loki in the Cloud, Winston fails to send logs to the right URL.

Cloud Logs uses authentication via an API key which is set in the URL
It looks like this:
https://12345:<api_key>@logs-prod-eu-west-0.grafana.net/loki/api/v1/push

The API Key gets stripped off because winston-loki uses just the hostname and path to form the URL. so the URL becomes:
https://logs-prod-eu-west-0.grafana.net/loki/api/v1/push

This happens here:

winston-loki/src/requests.js

Line 19 in 6c7b3a1

const options = {

Outcome
The correct link should be used.

Any plans to cut a new release?

First of all - thanks for this transport. I'm trying to use it to ship logs from our Lambda functions to our Loki instance and it looks like it's exactly what I need.
The current version in NPM is very different than the one currently in master. Any plans to cut a new release soon? Even an rc we can use to field test this module?

Support custom labels and line

The labels and the line should be customisable from outside of the transport.

winston-loki/index.js

Lines 24 to 32 in ca1af93

	labels: `{job="${label}", level="${level}"}`,
	entries: [
	{
	ts: timestamp,
	line: `${message} ${
	rest && Object.keys(rest).length > 0 ? JSON.stringify(rest) : ''
	}`
	}
	]

Add basicAuth in README.md

There is a basicAuth option as shown in this example.

However, this option does not appear in the README.md.

Related issue: #76

I can open a PR if you'd like 😄

Request - please cut a new version with `snappy` fix

Hi folks!

I wonder if it's possible you could cut a new release with the 7.* snappy version. Depending on winston-loki with the older snappy has brought in a vulnerable simple-get in our project, and it seems we need a newer version of snappy to avoid it.

The relevant dependency path we see is:

Exposure of Sensitive Information in simple-get - https://github.com/advisories/GHSA-wpg7-2c88-r8xv
fix available via `npm audit fix --force`
Will install [email protected], which is a breaking change
node_modules/prebuild-install/node_modules/simple-get
  prebuild-install  <=6.1.4
  Depends on vulnerable versions of simple-get
  node_modules/prebuild-install
    snappy  6.1.0 - 6.3.5
    Depends on vulnerable versions of prebuild-install
    node_modules/snappy
      winston-loki  >=2.0.0
      Depends on vulnerable versions of snappy
      node_modules/winston-loki

(here npm audit fix is suggesting we downgrade to an old version because of the dependency on snappy->prebuild-install.

Thanks!

not showing debug log in loki

I am using following code. I am seeing info level and error level but not debug level or logs what is the issue ?
logger.debug('I am a debug log')
logger.info('I am a info log')
logger.error('I am a error log')

Erroneous console.log in latest rc

There are a few debug logs left in the batcher

winston-loki/src/batcher.js

Lines 187 to 194 in 7492899

	console.log(message)
	// Encode the PushRequest object and create the binary buffer
	const buffer = logproto.PushRequest.encode(message).finish()
	console.log(buffer)
	// Compress the buffer with snappy
	reqBody = snappy.compressSync(buffer)
	} catch (err) {
	console.log(err)

the logs are not sent after "something" happened.

This is more an open question than an issue...

I have a couple of services that use winston-loki as transport to send the logs data to Loki. It's configured to send logs as batch (as default).

I noticed that sometimes (in moments where one of the services is a little bit "stressed" and received many messages) the transport simply stops to send logs to Loki, and therefore, I find no logs in there. The service is continuing to do stuff, as I can see that the Console transport of winston is still working, and displaying correctly things on the console.
The problem is that it never sends a log again, so I've lost even days of logs...

So my question is: what can be the reasons why the batching stopped sending logs? And when it stopped, how can I automatically make it restart sending the logs that it has stacked until that moment?

thank you,
Giacomo.

Enable `replaceTimestamp` by default

Changes in the upstream Loki project require us to activate the replaceTimestamp option, or otherwise, logs uploading will silently fail.
See #90 and #75

I suggest enabling this option by default and adding a warning in the option's description as to why users should not turn it off.

I can contribute a PR for this.
Thank you for your attention.

Checklist

• Modify index.js ✓ c0d23d2 Edit
• Running GitHub Actions for index.js ✓ Edit
• Modify README.md ✓ e279641 Edit
• Running GitHub Actions for README.md ✓ Edit

TypeError: winston_loki_1.default is not a constructor

Hello there, I am using this module with Nestjs. I have a problem

npm run start:dev                                                                                                                                       4s  14.20.1  17:15:07

> [email protected] start:dev /Users/binhvo/Workspaces/devpanel/uiv4backend-v2
> nest start --watch
[5:15:38 PM] Starting compilation in watch mode...

[5:15:57 PM] Found 0 errors. Watching for file changes.


/Users/binhvo/Workspaces/devpanel/uiv4backend-v2/src/app.module.ts:58
        new LokiTransport({
        ^
TypeError: winston_loki_1.default is not a constructor
    at Object.<anonymous> (/Users/binhvo/Workspaces/devpanel/uiv4backend-v2/src/app.module.ts:58:9)
    at Module._compile (internal/modules/cjs/loader.js:1085:14)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1114:10)
    at Module.load (internal/modules/cjs/loader.js:950:32)
    at Function.Module._load (internal/modules/cjs/loader.js:790:12)
    at Module.require (internal/modules/cjs/loader.js:974:19)
    at require (internal/modules/cjs/helpers.js:101:18)
    at Object.<anonymous> (/Users/binhvo/Workspaces/devpanel/uiv4backend-v2/src/main.ts:4:1)
    at Module._compile (internal/modules/cjs/loader.js:1085:14)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1114:10)

Avoid snappy dependency

It would be nice if this package could avoid the snappy dependency, because it requires native binaries to be built.

This is tricky if a project uses the alpine version of node Docker image for production builds. Sure, one could always install all the python dependencies to build those binaries, but it seems overkill.

Deprecated push endpoint used at the moment

Should be /loki/api/v1/push instead of /loki/prom/push

Fix broken tests

There's a bunch of tests that don't work, mostly because the fixtures in ./test/fixtures.json are invalid. This is because there's less data processing happening in 6.0.0 before sending than before. Logs in this version only get formatted and stringified just before sending out for easier processing.

Checklist

• Modify ./test/fixtures.json ✓ 83d36dd Edit
• Running GitHub Actions for ./test/fixtures.json ✓ Edit
• Modify ./test/transport.test.js ✓ 32ce710 Edit
• Running GitHub Actions for ./test/transport.test.js ✓ Edit
• Modify ./test/batcher.json.test.js ✓ 452f6a5 Edit
• Running GitHub Actions for ./test/batcher.json.test.js ✓ Edit
• Modify ./test/batcher.protobuf.test.js ✓ 64adbf7 Edit
• Running GitHub Actions for ./test/batcher.protobuf.test.js ✓ Edit
• Modify ./test/custom-labels-lines.test.js ✓ cab7928 Edit
• Running GitHub Actions for ./test/custom-labels-lines.test.js ✓ Edit
• Modify ./test/requests.test.js ! No changes made Edit
• Running GitHub Actions for ./test/requests.test.js ✗ Edit

Support custom labels

https://github.com/grafana/loki/blob/master/docs/logql.md#logql-log-query-language
Not only job=something, but also {app='apppppp', .... any string}.

Thanks

Shouldn't prepareProtoBatch be idempotent?

Hi, thanks for this package!

When running the transport in batch mode and without a functioning Loki instance. The loop keeps running prepareProtoBatch over and over the same batch. For example:

const { prepareProtoBatch } = require('../src/proto/helpers')

let batch = {
  streams: [
    {
      labels: { level: 'error', label: 'test' },
      entries: [
        {
          timestamp: { seconds: 1546977615, nanos: 848000000 },
          line: 'you broke everything'
        }
      ]
    }
  ]
}

for (const x of Array(3).keys()) {
  batch = prepareProtoBatch(batch)
  console.log(`-----${x+1}-----`)
  console.log(batch.streams[0].labels)
}

Results:

$ node test/proto.js
-----1-----
{level="error",label="test"}
-----2-----
{level="undefined",0="{",1="l",2="e",3="v",4="e",5="l",6="=",7=""",8="e",9="r",10="r",11="o",12="r",13=""",14=",",15="l",16="a",17="b",18="e",19="l",20="=",21=""",22="t",23="e",24="s",25="t",26=""",27="}"}
-----3-----
{level="undefined",0="{",1="l",2="e",3="v",4="e",5="l",6="=",7=""",8="u",9="n",10="d",11="e",12="f",13="i",14="n",15="e",16="d",17=""",18=",",19="0",20="=",21=""",22="{",23=""",24=",",25="1",26="=",27=""",28="l",29=""",30=",",31="2",32="=",33=""",34="e",35=""",36=",",37="3",38="=",39=""",40="v",41=""",42=",",43="4",44="=",45=""",46="e",47=""",48=",",49="5",50="=",51=""",52="l",53=""",54=",",55="6",56="=",57=""",58="=",59=""",60=",",61="7",62="=",63=""",64=""",65=""",66=",",67="8",68="=",69=""",70="e",71=""",72=",",73="9",74="=",75=""",76="r",77=""",78=",",79="1",80="0",81="=",82=""",83="r",84=""",85=",",86="1",87="1",88="=",89=""",90="o",91=""",92=",",93="1",94="2",95="=",96=""",97="r",98=""",99=",",100="1",101="3",102="=",103=""",104=""",105=""",106=",",107="1",108="4",109="=",110=""",111=",",112=""",113=",",114="1",115="5",116="=",117=""",118="l",119=""",120=",",121="1",122="6",123="=",124=""",125="a",126=""",127=",",128="1",129="7",130="=",131=""",132="b",133=""",134=",",135="1",136="8",137="=",138=""",139="e",140=""",141=",",142="1",143="9",144="=",145=""",146="l",147=""",148=",",149="2",150="0",151="=",152=""",153="=",154=""",155=",",156="2",157="1",158="=",159=""",160=""",161=""",162=",",163="2",164="2",165="=",166=""",167="t",168=""",169=",",170="2",171="3",172="=",173=""",174="e",175=""",176=",",177="2",178="4",179="=",180=""",181="s",182=""",183=",",184="2",185="5",186="=",187=""",188="t",189=""",190=",",191="2",192="6",193="=",194=""",195=""",196=""",197=",",198="2",199="7",200="=",201=""",202="}",203=""",204="}"}

Is this expected behavior?

Edit:

The following patch fixes it (didn't test it though):

diff --git src/proto/helpers.js src/proto/helpers.js
index 088bf15..b6210b4 100644
--- src/proto/helpers.js
+++ src/proto/helpers.js
@@ -28,6 +28,9 @@ module.exports = {
   },
   prepareProtoBatch: batch => {
     batch.streams = batch.streams.map(logEntry => {
+      if (typeof logEntry.labels === 'string') {
+        return logEntry
+      }
       let protoLabels = `{level="${logEntry.labels.level}"`
       delete logEntry.labels.level
       for (let key in logEntry.labels) {

With patch:

$ node test/proto.js
-----1-----
{level="error",label="test"}
-----2-----
{level="error",label="test"}
-----3-----
{level="error",label="test"}

Use GitHub actions for automatic releases

https://www.npmjs.com/package/semantic-release

Transport closes before logs are shipped

I'm trying to use this transport in an AWS Lambda function and I'm finding that the function terminates before the logs are shipped to Loki.
I've noticed this issue in Winston's main repository and tried to implement this workaround in order to force the function to wait for the logger to finish shipping the logs but it does not wait for the requests to be sent before resolving the promise and closing the function.

I suspect it has something to do with this line which immediately emits a logged event to winston instead of waiting for the log record to be pushed, but honestly I feel I'm not familiar enough with Winston to be sure this is exactly what's happening.

Could emitting the logged event further down the line help with the scenario I've presented?