metarhia / metalog Goto Github PK
View Code? Open in Web Editor NEWMetarhia logger ๐
Home Page: https://metarhia.com
License: MIT License
Metarhia logger ๐
Home Page: https://metarhia.com
License: MIT License
At the moment tests and development configs are published to npm.
@belochub multi-line warnings with stack break logs structure. Potentially we may log multi-line messages without stack. Please fix this or assign somebody. Example:
2018-12-21T19:41:00.394Z warn Possible EventEmitter memory leak detected. 11 ups listeners added. Use emitter.setMaxListeners() to increase limit MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 ups listeners added. Use emitter.setMaxListeners() to increase limit
at _addListener (events.js:243:17)
at Application.addListener (events.js:259:10)
at /applications/example/lib/application.test.js:33:15
at Script.runInContext (vm.js:107:20)
at prepareScript (/lib/scripts.js:42:28)
at api.fs.readFile (/lib/scripts.js:66:17)
at FSReqWrap.readFileAfterClose [as oncomplete] (internal/fs/read_file_context.js:53:3)
Support custom destination to write logs into external logging systems like database or api
Problem
Console interface not fully implements js Console interface.
debug, error, log, info, warn
contracts are different from js Console contracts.
https://developer.mozilla.org/en-US/docs/Web/API/console/log
It must handle 2 scenarios: multiple objects to log, or string as a first argument and objects to replace substitution parameters as rest.
Solution
For those methods - implement handling different arguments based from first argument type. It can be an object - rest are objects or a string - rest are objects to replace substitution parameters.
@tshemsedinov What are your thoughts about this?
Change console.log
to process.stdout.write
Right now all loggers take only one string as an argument. I propose to change that to conform to the console.{log,warn}
etc interface. This will improve interoperability of those methods and allow to omit unnecessary concatenations/templates when the appropriate logger is disabled altogether.
Logging is just a derivative of observation IMHO
https://opentelemetry.io/docs/concepts/data-collection/
for example the exporter libraries:
ps. https://www.itsumma.ru/knowledges/blog/monitoringisdead
api - https://github.com/open-telemetry/opentelemetry-proto
Sorry if off topic...
Handle errors for:
If logger can't write logs to files we need to pass error to upper modules and write everything to console. Don't use common.emptiness
stubs. Don't throw errors, just log to console.
Logger configuration/initialization should have a list of log types to output them to stdout formatted and styled with concolor
We need no this wrapper anymore, because each process in Metarhia technology stack will host single application in Impress application server 2.x (see metarhia/impress#1207)
This supersedes #74.
We will can use following syntax:
await logger.open();
await logger.close();
await metalog(options);
this will call return this.open();
from constructorwritableBuffer
https://github.com/metarhia/metalog/tree/writable-buffer-corkfs.open/write
https://github.com/metarhia/metalog/tree/use-fs-openProblem
We need to write some logs to a db (postgres/redis, etc.) and be able to write logs by meaning. Write operation log to a specific table in a database, write statistic logs to a different table, etc.
Solution
Provide writable destinations by LoggerOptions
from config file. Writable destination must have method write(buffer, fn)
similar as NodeJS Writable Stream. Logger must support multiple write destinations like a set of writables.
Impress logger has init
and open
. We will remove init
and implement auto-open from Logger constructor.
Flush buffer to disk after options.writeBuffer
exceeded
Implementation @machendos
Review @lundibundi
Problem
We need to have multiple high level log groups, which have meaning for our application such as: statistics, monitoring, telemetry, security, db operations, etc.
Solution
Extend existing logger functionality by adding new method: groupLog(message, groupName = 'system')
Use templates to concatenate more then two strings.
Need research with metarhia/Example
. Sometimes log files are empty but stdout shows logs.
After implementing metalog Impress will hold all logs in single directory and rotate logs just from master process.
Problem
Logger and file system writable destination now implemented together. For more good code quality we may move away logic, that create folders and write files to a separate module.
Solution
Extract logic for writing files from Logger and implement in a separate module without losing functionality. File writable destination may be optional as well.
Implement async/await interface in addition to events and callbacks.
const logger = await new Logger();
await logger.open();
await logger.close();
await logger.rotate();
Hello,
Have made mocha tests for metalog.
Once running tests like this:
mocha metalogtest.js | tee 1
I get this error:
1) metalog
init:
Uncaught TypeError: Cannot read property 'stack' of null
at fs.unlink.err (/home/someuser/Downloads/tests/node_modules/metalog/metalog.js:180:41)
at FSReqWrap.args [as oncomplete] (fs.js:140:20)
Another note that forwarding output streams using tee or &> 1 doesn't work either, the 1 output doesn't get created.
Here is quick fix:
################
rotate() {
if (!this.keepDays) return;
fs.readdir(this.path, (err, files) => {
if (err) {
process.stdout.write(`${err.stack}\n`);
this.emit('error', err);
return;
}
const now = new Date();
const year = now.getUTCFullYear();
const month = now.getUTCMonth();
const day = now.getUTCDate();
const date = new Date(year, month, day, 0, 0, 0, 0);
const time = date.getTime();
for (const fileName of files) {
const fileTime = new Date(fileName.substring(0, 10)).getTime();
const fileAge = Math.floor((time - fileTime) / DAY_MILLISECONDS);
if (fileAge > 1 && fileAge > this.keepDays - 1) {
fs.unlink(this.path + '/' + fileName, err => {
if (err) { // QUICK FIX
process.stdout.write(`${err.stack}\n`);
this.emit('error', err);
}
});
}
}
});
}
####################################
Here is my simple test itself:
describe('metalog', ()=>{
let logger;
after(()=>{
logger.close();
});
it('init', ()=>{
const metalog = require('metalog');
const config = {
enabled: true,
keepDays: 100, // Delete files after N days
writeInterval: 3000,//Duration('3s'), // Flush log to disk interval
writeBuffer: 64 * 1024, // Buffer size 64kb
toStdout: ['system', 'fatal', 'error']
};
const { writeInterval, writeBuffer, keepDays, toStdout, toFile } = config;
const logDir = process.cwd();
const nodeId = 'N1';
logger = metalog({
path: logDir,
node: nodeId,
writeInterval,
writeBuffer,
keepDays,
toStdout,
toFile
});
});
it('system', ()=>{
logger.system('1.system');
});
});
Describe the bug
When you disable the file system in the configuration by defining the 'toFile' param with an empty array, it still tries to create a log folder or read a log file from it.
To Reproduce
Expected behavior
The log folder doesn't create
Actual behavior
The log folder present at the root of the project
Desktop (please complete the following information):
At the moment if the process encounters an infinite loop it is possible to loose console.log
output, from the same sync
block.
I believe there should be a way to force process.stdout to write everything to he output right now.
Export object with keys to be compatible with other Metarhia libraries
Optimize buffering with writableBuffer
and cork/uncork
We call console.debug
from certain file so it will write file name to log
on('open', () => {})
on('close', () => {})
on('rotate', () => {})
on('error', (err) => {})
Now we have stdout format like this:
2018-12-23T22:26:31.539Z system Worker forked
2018-12-23T22:26:31.539Z <type> <Message>
and file logs looks like:
2018-12-23T22:36:28.822Z [system] Worker forked
2018-12-23T22:36:28.822Z [<type>] <Message>
But it will be great to improve output.
Formatting for stdout:
22:26:31 system S1N0/impress Worker forked
22:26:31 <type> <process>/<application> <Message>
and file logs:
2018-12-23T22:36:28.822Z [system] S1N0/impress Worker forked
2018-12-23T22:36:28.822Z [<type>] <process>/<application> <Message>
@belochub @nechaido @lundibundi please assign somebody to this
We have reopen
implementation but I thin it isn't working after moving from Impress
Remove parentheses around single argument arrow functions
In metalog
unlike old Impress logger you cant configure custom log files or new custom log levels, everything will be fixed, see #4 for detains.
master
- everything about forking, stopping and lifecycle, process healthworker
- everything about forking, stopping and lifecycle, process healthaccess
- http, jstp, metarhia protocol or other protocol (attached to certain process)slow
- slow requests for http, jstp, metarhia protocol, etc.Each file will contain following levels: error, warn, info, debug
Rename node
option to workerId
option.
New option Writable
(default to WritableFileStream
from metastreams
)
We don't need application
option (see #98)
Format stack trace for logger.fatal
and logger.error
We have 2 log source types: M - Master, W - Worker
We have 2 logger types: S - Server logs (single per process), A - Application Logs (for each app)
So we have combination: {M,S}, {M,A} (application starts just in workers), {W,S}, {W,A}
/log/YYYY-DD-MM-NODEID-impress.log
, examples:
/log/2017-10-04-S1N0-impress.log
/log/YYYY-DD-MM-NODEID-impress.log
, examples:
/log/2017-10-04-S1N1-impress.log
/applications/name/log/YYYY-DD-MM-NODEID-TYPE.log
, examples:
/log/2017-10-04-S1N1-metacom.log
/log/2017-10-04-S1N1-serv.log
/log/2017-10-04-S1N1-slow.log
If toFile: []
is empty array, so we will not log anything to file, in this case we need not even open file stream and call rotation on timer, fetch buffers and so on. In short we need console-only logger with minimal config. Following will work but file operations will be still executed:
const logger = metalog({
node: 'S1N1',
application: 'app1',
toFile: [],
});
We need it in impress, to use for logging before load configuration. For this case metalog()
will synchronously call Logger.open()
and it will synchronously emit open
event (not it doesn't) as well as close
event should be emitted immediately/synchronously on Logger.close()
.
message.length <= buffer.freeSize
: Add message to buffer, do not emit data
, start timeoutmessage.length > buffer.freeSize
: Concatenate message and buffer, emit, remove timeoutA declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.