metarhia/metalog

Metarhia logger 📝

Meta Logger for Metarhia

Output example

Usage

const logger = await metalog.openLog({
  path: './log', // absolute or relative path
  workerId: 7, // mark for process or thread
  writeInterval: 3000, // flush log to disk interval
  writeBuffer: 64 * 1024, // buffer size (default 64kb)
  keepDays: 5, // delete after N days, 0 - disable
  home: process.cwd(), // remove substring from paths
  json: false, // print logs in JSON format, by default false
});

const { console } = logger;
console.log('Test message');
await logger.close();

License & Contributors

Copyright (c) 2017-2023 Metarhia contributors. Metalog is MIT licensed.
Metalog is a part of Metarhia technology stack.

metalog's Issues

Prepare load test

Reduce log files

• Master:
  • master - everything about forking, stopping and lifecycle, process health
• Worker:
  • worker - everything about forking, stopping and lifecycle, process health
  • access - http, jstp, metarhia protocol or other protocol (attached to certain process)
  • slow - slow requests for http, jstp, metarhia protocol, etc.

Each file will contain following levels: error, warn, info, debug

Return promises from open and close methods

We will can use following syntax:

• await logger.open();
• await logger.close();
• await metalog(options); this will call return this.open(); from constructor

Fetch buffer to file on process crash

Need research with metarhia/Example. Sometimes log files are empty but stdout shows logs.

Use Writable.writableBuffer instead of Logger.buffer

Optimize buffering with writableBuffer and cork/uncork

Multi-line logging for warn, info, assess, slow and system

@belochub multi-line warnings with stack break logs structure. Potentially we may log multi-line messages without stack. Please fix this or assign somebody. Example:

2018-12-21T19:41:00.394Z	 warn   	Possible EventEmitter memory leak detected. 11 ups listeners added. Use emitter.setMaxListeners() to increase limit MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 ups listeners added. Use emitter.setMaxListeners() to increase limit
    at _addListener (events.js:243:17)
    at Application.addListener (events.js:259:10)
    at /applications/example/lib/application.test.js:33:15
    at Script.runInContext (vm.js:107:20)
    at prepareScript (/lib/scripts.js:42:28)
    at api.fs.readFile (/lib/scripts.js:66:17)
    at FSReqWrap.readFileAfterClose [as oncomplete] (internal/fs/read_file_context.js:53:3)

Update to metatests 0.2.4

Implementation @machendos
Review @lundibundi

Support multiple write destinations by logger.

Problem

We need to write some logs to a db (postgres/redis, etc.) and be able to write logs by meaning. Write operation log to a specific table in a database, write statistic logs to a different table, etc.

Solution

Provide writable destinations by LoggerOptions from config file. Writable destination must have method write(buffer, fn) similar as NodeJS Writable Stream. Logger must support multiple write destinations like a set of writables.

Implement async/await interface

Implement async/await interface in addition to events and callbacks.

• Constructor const logger = await new Logger();
• Open await logger.open();
• Close await logger.close();
• Rotate await logger.rotate();

An issue with disabling the file system

Describe the bug
When you disable the file system in the configuration by defining the 'toFile' param with an empty array, it still tries to create a log folder or read a log file from it.

To Reproduce

1. Delete the 'log' folder from the root of the project or ensure it's not there;
2. Set the 'toFile' property with an empty array inside of config;
3. Run the program.

Expected behavior
The log folder doesn't create

Actual behavior
The log folder present at the root of the project

Desktop (please complete the following information):

• OS: macOS 13.1
• Node.js version 16.15.1

Optimize console-only logger

If toFile: [] is empty array, so we will not log anything to file, in this case we need not even open file stream and call rotation on timer, fetch buffers and so on. In short we need console-only logger with minimal config. Following will work but file operations will be still executed:

const logger = metalog({
  node: 'S1N1',
  application: 'app1',
  toFile: [],
});

We need it in impress, to use for logging before load configuration. For this case metalog() will synchronously call Logger.open() and it will synchronously emit open event (not it doesn't) as well as close event should be emitted immediately/synchronously on Logger.close().

Remove unnecessary files from npm package

At the moment tests and development configs are published to npm.

Create log folder if not exists

Deprecate custom logs

In metalog unlike old Impress logger you cant configure custom log files or new custom log levels, everything will be fixed, see #4 for detains.

Define log types for stdout

Logger configuration/initialization should have a list of log types to output them to stdout formatted and styled with concolor

Use events instead of callbacks

• Event: on('open', () => {})
• Event: on('close', () => {})
• Event: on('rotate', () => {})
• Event: on('error', (err) => {})

Convert to class syntax

• Convert static (class) methods to simple functions

Changes Logger constructor options

Rename node option to workerId option.
New option Writable (default to WritableFileStream from metastreams)
We don't need application option (see #98)

Use string operations instead of regular expressions

Add `access` logging level

This supersedes #74.

/cc @nechaido @o-rumiantsev

Format stack trace

Format stack trace for logger.fatal and logger.error

Change loggers API to conform to `console` methods

Right now all loggers take only one string as an argument. I propose to change that to conform to the console.{log,warn} etc interface. This will improve interoperability of those methods and allow to omit unnecessary concatenations/templates when the appropriate logger is disabled altogether.

/cc @tshemsedinov @nechaido @belochub

Create new logs each day

We have reopen implementation but I thin it isn't working after moving from Impress

Log rotation

After implementing metalog Impress will hold all logs in single directory and rotate logs just from master process.

Compare implementations

• Manual buffering https://github.com/metarhia/metalog/tree/from-impress
• Use writableBuffer https://github.com/metarhia/metalog/tree/writable-buffer-cork
• Use fs.open/write https://github.com/metarhia/metalog/tree/use-fs-open

How about support for the OpenTelemetry concept?

Logging is just a derivative of observation IMHO
https://opentelemetry.io/docs/concepts/data-collection/

for example the exporter libraries:

• Console
• In-memory
• Jaeger
• Zipkin
• Prometheus

ps. https://www.itsumma.ru/knowledges/blog/monitoringisdead
api - https://github.com/open-telemetry/opentelemetry-proto

Sorry if off topic...

Fix single argument functions

Remove parentheses around single argument arrow functions

Use stdout to write to sonsole

Change console.log to process.stdout.write

Support full log contract from plain js Console in metalog.Console

Problem

Console interface not fully implements js Console interface.

debug, error, log, info, warn contracts are different from js Console contracts.
https://developer.mozilla.org/en-US/docs/Web/API/console/log

It must handle 2 scenarios: multiple objects to log, or string as a first argument and objects to replace substitution parameters as rest.

Solution

For those methods - implement handling different arguments based from first argument type. It can be an object - rest are objects or a string - rest are objects to replace substitution parameters.

@tshemsedinov What are your thoughts about this?

using tee for redirection leads to error

Hello,

Have made mocha tests for metalog.
Once running tests like this:

mocha metalogtest.js | tee 1

I get this error:

  1) metalog
       init:
     Uncaught TypeError: Cannot read property 'stack' of null
      at fs.unlink.err (/home/someuser/Downloads/tests/node_modules/metalog/metalog.js:180:41)
      at FSReqWrap.args [as oncomplete] (fs.js:140:20)

Another note that forwarding output streams using tee or &> 1 doesn't work either, the 1 output doesn't get created.

Here is quick fix:
################

  rotate() {
    if (!this.keepDays) return;
    fs.readdir(this.path, (err, files) => {
      if (err) {
        process.stdout.write(`${err.stack}\n`);
        this.emit('error', err);
        return;
      }
      const now = new Date();
      const year = now.getUTCFullYear();
      const month = now.getUTCMonth();
      const day = now.getUTCDate();
      const date = new Date(year, month, day, 0, 0, 0, 0);
      const time = date.getTime();
      for (const fileName of files) {
        const fileTime = new Date(fileName.substring(0, 10)).getTime();
        const fileAge = Math.floor((time - fileTime) / DAY_MILLISECONDS);
        if (fileAge > 1 && fileAge > this.keepDays - 1) {
          fs.unlink(this.path + '/' + fileName, err => {
if (err) { // QUICK FIX
            process.stdout.write(`${err.stack}\n`);
            this.emit('error', err);
}
          });
        }
      }
    });
  }

####################################
Here is my simple test itself:

describe('metalog', ()=>{
    let logger;

    after(()=>{
        logger.close();
    });

    it('init', ()=>{
        const metalog = require('metalog');

        const config = {
            enabled: true,
            keepDays: 100, // Delete files after N days
            writeInterval: 3000,//Duration('3s'), // Flush log to disk interval
            writeBuffer: 64 * 1024, // Buffer size 64kb
            toStdout: ['system', 'fatal', 'error']
        };

        const { writeInterval, writeBuffer, keepDays, toStdout, toFile } = config;

        const logDir = process.cwd();
        const nodeId = 'N1';
        logger = metalog({
            path: logDir,
            node: nodeId,
            writeInterval,
            writeBuffer,
            keepDays,
            toStdout,
            toFile
        });
    });

    it('system', ()=>{
        logger.system('1.system');
    });
});

Support Logging before Infinite Loop

At the moment if the process encounters an infinite loop it is possible to loose console.log output, from the same sync block.

I believe there should be a way to force process.stdout to write everything to he output right now.

Decompose Logger, move away hard coded folder-file writable source

Problem

Logger and file system writable destination now implemented together. For more good code quality we may move away logic, that create folders and write files to a separate module.

Solution

Extract logic for writing files from Logger and implement in a separate module without losing functionality. File writable destination may be optional as well.

Remove ApplicationLogger wrapper

We need no this wrapper anymore, because each process in Metarhia technology stack will host single application in Impress application server 2.x (see metarhia/impress#1207)

Support options.writeBuffer

Flush buffer to disk after options.writeBuffer exceeded

Implement `groupLog` method to support log groups by providing metadata

Problem

We need to have multiple high level log groups, which have meaning for our application such as: statistics, monitoring, telemetry, security, db operations, etc.

Solution

Extend existing logger functionality by adding new method: groupLog(message, groupName = 'system')

Change export contract

Export object with keys to be compatible with other Metarhia libraries

New logging format

Now we have stdout format like this:

2018-12-23T22:26:31.539Z	 system 	Worker forked
2018-12-23T22:26:31.539Z	 <type> 	<Message>

and file logs looks like:

2018-12-23T22:36:28.822Z	[system]	Worker forked
2018-12-23T22:36:28.822Z	[<type>]	<Message>

But it will be great to improve output.
Formatting for stdout:

22:26:31  system  S1N0/impress  Worker forked
22:26:31  <type>  <process>/<application>  <Message>

and file logs:

2018-12-23T22:36:28.822Z [system] S1N0/impress Worker forked
2018-12-23T22:36:28.822Z [<type>] <process>/<application> <Message>

@belochub @nechaido @lundibundi please assign somebody to this

Support JSON log serialization

WriteBuffer

• If message.length <= buffer.freeSize : Add message to buffer, do not emit data, start timeout
• If message.length > buffer.freeSize : Concatenate message and buffer, emit, remove timeout

Handle error

Handle errors for:

• fs.unlink
• fs.stats
• fs.readdir

If logger can't write logs to files we need to pass error to upper modules and write everything to console. Don't use common.emptiness stubs. Don't throw errors, just log to console.

Change string concatenation to template strings

Use templates to concatenate more then two strings.

Support custom destination

Support custom destination to write logs into external logging systems like database or api

Log files naming

We have 2 log source types: M - Master, W - Worker
We have 2 logger types: S - Server logs (single per process), A - Application Logs (for each app)

So we have combination: {M,S}, {M,A} (application starts just in workers), {W,S}, {W,A}

• {M,S} -> /log/YYYY-DD-MM-NODEID-impress.log, examples:
  • /log/2017-10-04-S1N0-impress.log
• {W,S} -> /log/YYYY-DD-MM-NODEID-impress.log, examples:
  • /log/2017-10-04-S1N1-impress.log
• {W,A} -> /applications/name/log/YYYY-DD-MM-NODEID-TYPE.log, examples:
  • /log/2017-10-04-S1N1-metacom.log
  • /log/2017-10-04-S1N1-serv.log
  • /log/2017-10-04-S1N1-slow.log

Log path for console.debug

We call console.debug from certain file so it will write file name to log

Refactor init/open

Impress logger has init and open. We will remove init and implement auto-open from Logger constructor.