platformlab/nanolog

Nanolog is an extremely performant nanosecond scale logging system for C++ that exposes a simple printf-like API.

nanolog's Issues

ability to file swap

Is there capability such that the file size does not very large? Perhaps some ability to roll the file once it becomes 10GB in size?

Signal Catching

do you plan to incorporate signal catching into NanoLog? Would be great to have them automatically caught and logged rather than having to deduce what went wrong, then instrument with logging, then rerun... rinse repeat. And/or to avoid having to use another signal logging library.

Strictly Single Threaded Usage

The applications I'm going to logging are strictly single threaded and pinned to logical CPUs. And I read in the documentation that a background thread is used for dumping the logged data to disk. Is there anyway to... not use a background thread and just periodically dump the log myself?

Structured Hierarchical Log Output

have you considered adding hierarchy and structure into the logging?

For example to record a function call stack to implement tracing. Like a right tab for each level deeper, and a left tab as you progress out.

With LLTng at 90ns per event (https://youtu.be/qKqGo4FFPx0?t=402) it's only logical to want NanoLog to be my tracer as well lol.

Comparisson between preprocessor and C++17

Hello,

Could you please update the ReadMe with the expected performance difference(in your opinion) between the pre-processor version and the C++17 one? Thank you very much

Can authors write guides to introduce how we can use this project in the Windows environment?

Can authors make a instruction so as to guide other programmers to use tihis project in the Windows environment?

Performance

NanoLogCpp17 conflicts with Boost

I use NanoLog with and a bunch of boost libraries in my project, and I find that in debug mode there seem to be conflicts in assert definition. The compiler gives me errors like

error: ‘__assert_fail’ was not declared in this scope

I also find that if I include NanoLogCpp17.h after the boost libraries, the error will be gone. I don't know the root cause of this issue but it looks like a macro definition issue to me. Here is a minimal example to demonstrate the problem:

#define BOOST_TEST_MODULE DEFS_TEST

#include "NanoLogCpp17.h"

#include <boost/test/included/unit_test.hpp>


BOOST_AUTO_TEST_SUITE(nanolog_boost)

BOOST_AUTO_TEST_CASE(dummy_test) {
    BOOST_CHECK(true);
}

BOOST_AUTO_TEST_SUITE_END()

Compile command: g++ test.cpp -I ~/NanoLog/runtime -I /usr/local/include/boost -L ~/NanoLog/runtime -lNanoLog -lrt -pthread
Result:

In file included from /usr/local/include/boost/assert.hpp:58:0,
                 from /usr/local/include/boost/smart_ptr/scoped_ptr.hpp:14,
                 from /usr/local/include/boost/scoped_ptr.hpp:13,
                 from /usr/local/include/boost/test/tree/fixture.hpp:20,
                 from /usr/local/include/boost/test/tree/decorator.hpp:22,
                 from /usr/local/include/boost/test/tree/test_unit.hpp:21,
                 from /usr/local/include/boost/test/impl/compiler_log_formatter.ipp:25,
                 from /usr/local/include/boost/test/included/unit_test.hpp:18,
                 from test.cpp:5:
/usr/local/include/boost/test/impl/junit_log_formatter.ipp: In member function ‘std::__cxx11::list<std::__cxx11::basic_string<char> > boost::unit_test::output::junit_result_helper::build_skipping_chain(const boost::unit_test::test_unit&) const’:
/usr/local/include/boost/test/impl/junit_log_formatter.ipp:187:9: error: ‘__assert_fail’ was not declared in this scope
         assert(m_map_test.count(tu.p_id) == 0 || results_collector.results( tu.p_id ).p_skipped);
         ^
/usr/local/include/boost/test/impl/junit_log_formatter.ipp:187:9: note: suggested alternative:
In file included from /usr/include/c++/7/cassert:44:0,
                 from /home/jie/NanoLog/runtime/Common.h:47,
                 from /home/jie/NanoLog/runtime/NanoLogCpp17.h:25,
                 from test.cpp:3:
/usr/include/assert.h:69:13: note:   ‘NanoLogInternal::__assert_fail’
 extern void __assert_fail (const char *__assertion, const char *__file,
             ^~~~~~~~~~~~~
In file included from /usr/local/include/boost/assert.hpp:58:0,
                 from /usr/local/include/boost/smart_ptr/scoped_ptr.hpp:14,
                 from /usr/local/include/boost/scoped_ptr.hpp:13,
                 from /usr/local/include/boost/test/tree/fixture.hpp:20,
                 from /usr/local/include/boost/test/tree/decorator.hpp:22,
                 from /usr/local/include/boost/test/tree/test_unit.hpp:21,
                 from /usr/local/include/boost/test/impl/compiler_log_formatter.ipp:25,
                 from /usr/local/include/boost/test/included/unit_test.hpp:18,
                 from test.cpp:5:
/usr/local/include/boost/test/impl/junit_log_formatter.ipp: In member function ‘virtual void boost::unit_test::output::junit_log_formatter::test_unit_finish(std::ostream&, const boost::unit_test::test_unit&, long unsigned int)’:
/usr/local/include/boost/test/impl/junit_log_formatter.ipp:551:5: error: ‘__assert_fail’ was not declared in this scope
     assert( tu.p_id == list_path_to_root.back() );
     ^
/usr/local/include/boost/test/impl/junit_log_formatter.ipp:551:5: note: suggested alternative:
In file included from /usr/include/c++/7/cassert:44:0,
                 from /home/jie/NanoLog/runtime/Common.h:47,
                 from /home/jie/NanoLog/runtime/NanoLogCpp17.h:25,
                 from test.cpp:3:
/usr/include/assert.h:69:13: note:   ‘NanoLogInternal::__assert_fail’
 extern void __assert_fail (const char *__assertion, const char *__file,
...

However, if you switch the order of including nanolog and boost, it would be successfully compiled.

Can I build a python wrap for nanolog?

Hi! I'd like to use nanolog in my python project, so I use pybind11 to wrap C++ into *.so that could be imported by python directly. Can I further make this into a python package and upload to pip?

Timestamp is not in order

The issue is that the timestamp is not in the right order. The time stamp of the log at 59 line is older than the timestamp of the log at 64 line. I only commented out the runBenMark() function.

~/NanoLog/sample$ ./decompressor decompress /tmp/logFile
2020-07-16 18:12:41.-00022964 main.cc:59 NOTICE[0]: A string, pointer, number, and float: 'Hello World', 0x7ffd9db8d8bc, 512, 3.141590
2020-07-16 18:12:41.-00022108 main.cc:64 NOTICE[0]: Shortend String: 'Hello World' and shortend float 3.14

Decompression Complete after printing 2 log messages

~/NanoLog/sample$ git diff
diff --git a/sample/main.cc b/sample/main.cc
index af67390..c4523cb 100644
--- a/sample/main.cc
+++ b/sample/main.cc
@@ -64,7 +64,7 @@ int main(int argc, char** argv) {
3.14159);

• runBenchmark();

• //runBenchmark();

  // Optional: Flush all pending log messages to disk
  NanoLog::sync();

MSVC Support?

Hello! Are you all going to end up supporting MSVC? I would absolutely love to use this library, but I have zero experience with gcc, and MSVC is what I always use!

Add a switch to enable log output to stdout.

No CMake integration

No CMake integration

Log with a single "%" cannot compile

Hi,

Everything else as usual, the following code

NANO_LOG(NOTICE, "%")

will cause compilation errors. I believe this is due to the fact that the syntax analyzer uses "%" as an indicator of log parameters, and here is complaining the missing of variable.

I'm not sure this is a feature or a bug, but in stdio, printf("%") is valid.

I admit that I found this phenomenon by chance (I forgot to delete a '%' when modifying my program), but if it is not of your expectation, maybe you can consider supporting outputing a single "%" character? (Nevertheless, it may add difficulty to the syntax analyzer I guess)

btw, I like you project so much! It helps me a lot.

Best,
bjmiao

how to make multiple logs at the same time

No conan or vcpkg integration

No conan or vcpkg integration

Try std::string_view instead

Hello,
Your logger implementation is really fast! Although we could not achieve the same performance as written in the documentation (maybe because of the difference in platform), it indeed outperform other loggers by far.

Now, I would like to make a small suggestion. I was trying to encapsulate the library inside of a class. Because of NanoLog's heavy dependency on constexpr, it was kind of cumbersome (just little tweaks here and there). To deal with the problem I changed the format string argument type from (&)[N] to std::string_view and it was easier to achieve my goal, without affecting performance.

My suggestion is just that: change (&)[N] to std::string_view.
With that in mind, I don't know if there is a better way of doing this, but using string_views is definitely a solution.

Thanks,
Kevin Baez.

build error

g++ -std=c++17 -g -O3 -DPREPROCESSOR_NANOLOG -I. -ItestHelper -I"../googletest/googletest"/include -c Cycles.cc -o Cycles.o
g++ -std=c++17 -g -O3 -DPREPROCESSOR_NANOLOG -I. -ItestHelper -I"../googletest/googletest"/include -c Util.cc -o Util.o
g++ -std=c++17 -g -O3 -DPREPROCESSOR_NANOLOG -I. -ItestHelper -I"../googletest/googletest"/include -c testHelper/GeneratedCode.cc -o testHelper/GeneratedCode.o
g++ -std=c++17 -g -O3 -DPREPROCESSOR_NANOLOG -I. -ItestHelper -I"../googletest/googletest"/include -c Log.cc -o Log.o
g++ -std=c++17 -g -O3 -DPREPROCESSOR_NANOLOG -I. -ItestHelper -I"../googletest/googletest"/include -c NanoLog.cc -o NanoLog.o
g++ -std=c++17 -g -O3 -DPREPROCESSOR_NANOLOG -I. -ItestHelper -I"../googletest/googletest"/include -c RuntimeLogger.cc -o RuntimeLogger.o
g++ -std=c++17 -g -O3 -DPREPROCESSOR_NANOLOG -I. -ItestHelper -I"../googletest/googletest"/include -c TimeTrace.cc -o TimeTrace.o
g++ -std=c++11 -isystem "../googletest/googletest"/include -I"../googletest/googletest" -c "../googletest/googletest"/src/gtest-all.cc -L. -lgtest -lrt -pthread
g++: error: ../googletest/googletest/src/gtest-all.cc: No such file or directory
GNUmakefile:44: recipe for target 'libgtest.a' failed
make: *** [libgtest.a] Error 1

No mention of C++17 no-preprocessor version in README

There is no mention of the C++17 no-preprocessor version of NanoLog in README. I stumbled upon it by accident when looking at commit log. Before that I was sure this library required integrating the preprocessor into your build system. What adds to the confusion is the still open pull request adding that functionality.

Transmit Binary Information over UDP!

Hi All,

Instead of logging locally I am planning to stream it over network. What I would like to know is, since this is a binary information if I pickup a truncated file will I loose all the information ?. If so can this be modified to extract the logs till which its written ?.

rgds
Balaji Kamal Kannadassan

No license information

What license does this project have? Can it be used in other open source or commercial projects?

I haven't found any information about it in the README.md. Individual files seem to either have some license text or none at all.

Can you guys provide some info how this can be ported to support other language like C#?

We are very interested to use NanoLog in our project but we are not C++ project.

Capability of file rolling

Hi, @syang0
We're going to use nanolog as our logging library, but we found that the file used for logging will never be rolled over. This makes it hard for a server to continuously writing logs to a file without the log file to be too large.
It is not convenient to use decompressor for each compressed log file to generate human readable log files. So can you provide any insight?
Thanks a lot!

Write to multiple log files.

Hi,

We are building C++ application where we need to log to multiple files. Pleas let me know how i can achieve this using NanoLog.

Thanks

Add a macro switch for a more convenient uncompressed output.

Why a "%d" specifier next to another "%d" specifier would crash the decompressor

Make modifications to "sample/main.cc" as shown in the figure below，

Then compile and run the program,

$ cd sample
$ make
$ ./sampleApplication
$ ./decompressor decompress compressedLog

What I expected to output is "My number is 987654321123456789", but the actual output is "My number is 987654321987654321". It outputs the integer 987654321 twice but doesn't output the integer 123456789.
On top of that, the decompressor aborted.

adding a comma between the two "%d" specifiers could solve the problem,

I'm confusing and wondering whether this is a bug or not.

Impossible to disable compression by default

Hello!

First of all, thank you very much for such a compact and fast logging library.

It would be nice to have a way to disable compression of logs by default. This will make the library more useful because right now it's impossible to use it if you want such features like:

• logs-based monitoring (filebeat, logstash,...) usually requires the log files to be parsable line by line.
• logging with huge amounts of data. Right now it's impossible to rotate log files (as far as I understand the library). Disabled compression will allow users to split files manually or with some help of additional tools.

Question: are you planning to add a parameter to disable compression by default? If not, please, at least give us some hints on how to do this properly - we'll try to do this on our own.

no such file or dir #include <aio.h>

MingW32-64: GCC 7.3

cyclesDiskIO_upperBound

NanoLog/runtime/RuntimeLogger.cc

Line 579 in dc0efa1

cyclesDiskIO_upperBound += (cyclesAtLastAIOStart - start);

maybe should be

cyclesDiskIO_upperBound += (start - cyclesAtLastAIOStart); 

possible to specify log filenames?

also, is it possible to let each thread write to its own log file?
when a log file becomes huge, can I restart a new log file?

I want to adopt this for a server app, which will run for months. Having a huge log file with all threads' outputs mixed together isn't convenient for processing.

Thanks.

Any suggestion to improve NanoLog type safety?

I learned in the hard way that NanoLog can behave very tricky when you have a typo in the format string, and the decompressor may not pick up properly from the encoded byte stream.

Here is one example:
NANO_LOG(NOTICE, "Test1 v=%4.2f", 4);
the above will print out "Test1 v=0.00"

Here is another one:
const char * str = "Hello World"; NANO_LOG(NOTICE, "Test2 v=%4.2f s=%.*s", 4, 5, str);
This will trigger assert to abort. If I change (*in) += sizeof(T); in unpack function for floating point values in Packer.h to (*in) += bytes;, then it will not abort and print out "Test2 v=0.00 s=Hello".

First question: is the change above in unpack function correct? Since unpack reads "bytes" of bytes from the memory to the destination variable, it makes sense to forward the memory location by bytes.

Second question/comment:
There may be a lot of other similar cases out there.
One suggestion is: right now, you only differentiate STRING from NON_STRING. Any chance you can take one step further and extend ParamType to tell different value type from NON_STRING, such as whether the non-string is an integer or a float? In this way, you can convert the passed in argument to the right value type (based on ParamType) in the store_argument function before you store the value.

Thank you.

Online decompressor

Is it supported or is it possible to implement: online decompressor (that not runs on an entire file and can be used while we logging)? E.G to support streaming online updates to some UI

sampleApplication in sample directory has segfault

I check out the head (4d1afc8) and then run make first in the runtime and then in the sample directory.
I then run ./sampleApplication in the sample directory, and it has segfault.
gdb indicates that the segfault happens in the compression thread.

TODO(syang0) Make this into a spin lock

Hello,
In runtime/RuntimeLogger.h I see this "todo", so I want to share this link where Linus Torvalds explains some pitfalls of spin locks in userspace, and why using them may be a bad idea.

system level use of nanlog?

How do you use nanolog in a system? Would you need to have one decompressor per binary? It would be good if there could be one decompressor for all applications in a system.

Run into "Internal Error: Corrupted BufferExtend"

The decompressor occassionally running into this error. If restart the decompressor (decompressUnordered mode) can continue without the error in the same binary log location.

But it could happen again later, this is killing the quality in production by dumping Text log file.

The error reported from log.cc #2022.

                fprintf(stderr, "Internal Error: Corrupted BufferExtent\r\n");

Has anyone encountered this? Any helpful hints are appreciated.

Nanosecond in formatted output could be negtive and log time is not well synced with system time

The initial logs just after program started always have a nanosecond part above or below 0 by a small value, and the calibrated cyclesPerSecond is not precise enough.

Actually there's a simple and precise way to convert a tsc to system nanosecond timestamp: tscns.

how about in multi-thread environment

How to tune NanoLog to better handle burst?

I'm testing NanoLog in a program, and noticed that roughly half of the messages were dropped in the log.

Here is the setting of my testing program. The program is multithreaded, and one of the worker threads is reading some msg (roughly 0.5k bytes per msg) from socket and log it and then process it. During the test, I read roughly ~160 msgs within 2 min from the socket, and logged all of them. However, only ~80 of them showed up in the log file. One of the cases I noticed is that the program handled ~10 msgs within 2 mini second, and only the first and last of the 10 msgs were successfully logged.

If I tweak the program to sleep for 1 second or call NanoLog::sync before logging the msg, all ~160 messages will be logged successfully.

So I looked at the runtime/Config.h file, and tried the following two settings, but strangely enough I saw no improvement and still roughly half of the messages were dropped.
STAGING_BUFFER_SIZE = 1<<25
RELEASE_THRESHOLD = 1<<19

STAGING_BUFFER_SIZE = 1<<25
RELEASE_THRESHOLD = 1<<20

Any suggestions on how I should tune the parameters to better handle burstiness? Thank you.

No Python 3 support

From looking at the makefiles and README it seems NanoLog supports only Python 2, which will lose all support at the end of 2019 - https://pythonclock.org. Because of that can you add support for Python 3?

Outdated benchmarks

README contains comparisons with other logging libraries without stating their versions - except for Boost 1.55, which came out 5 years ago in 2013. I presume other libraries in your tests are also as outdated as Boost.Log.

Can you update the benchmarks to reflect the current performance of the logging systems?

gcc-8 incompatibility - One of the threads gets stuck

Hello,

I might be doing something wrong but i have tried this on 3 different machines, 2 of them freshly installed.

Running the benchmark on a fresh installed Red Hat Enterprise Linux Server release 7.6 (Maipo) with devtoolset-8 and one of the threads gets stuck on RuntimeLogger::StagingBuffer::reserveSpaceInternal.
Have the same problem also on self-compiled gcc 8.2 install with ABI=1. (bin-utils =the ones from the devtoolset-8)

I also think that one of the threads crashes as there are only 2 live threads. Below the gdb bt and thread info. This is also a bit peculiar as it doesn't always happen, but it happens consistently enough (one every 4 runs let's say)

(gdb) bt
#0  NanoLogInternal::RuntimeLogger::StagingBuffer::reserveSpaceInternal (this=0x7ffb358ba010, nbytes=nbytes@entry=16, blocking=blocking@entry=true) at ../runtime/RuntimeLogger.cc:760
#1  0x000000000040513f in NanoLogInternal::RuntimeLogger::StagingBuffer::reserveProducerSpace (nbytes=16, this=<optimized out>) at ../runtime/RuntimeLogger.h:330
#2  NanoLogInternal::RuntimeLogger::reserveAlloc (nbytes=16) at ../runtime/RuntimeLogger.h:109
#3  __syang0__fl__Simple32log32message32with32032parameters__Benchmark46cc__74__ (level=NanoLog::LogLevels::NOTICE, fmtStr=<synthetic pointer>) at generatedCode.h:11
#4  runBenchmark (id=0, barrier=<optimized out>) at Benchmark.cc:74
#5  0x0000000000404b2b in main (argc=<optimized out>, argv=<optimized out>) at Benchmark.cc:119

(gdb) info threads
  Id   Target Id                                     Frame
* 1    Thread 0x7ffb359fe780 (LWP 47837) "benchmark" NanoLogInternal::RuntimeLogger::StagingBuffer::reserveSpaceInternal (this=0x7ffb358ba010, nbytes=nbytes@entry=16, blocking=blocking@entry=true) at ../runtime/RuntimeLogger.cc:760
  2    Thread 0x7ffb2c7d8700 (LWP 47840) "benchmark" 0x00007ffb34bb3d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

Thank you very much. will also keep an eye to provide more info if needed.

Failing Integration Tests

See attached error log when running integrationTest/run.sh on Linux RHEL 7.4 with gcc 8.3.1 and python 2.7.5.

IntegerationTestsErrors.txt

Having a background text file writer instead of the compressor with binary log writer

NanoLog is a very impressive logger that does what many other loggers don't quite achieve, like writing logged data directly into string buffers instead of using various intermediate variables, or capturing time stamps at the time of logging instead of at the time of writing into a log sink to ensure their order (even though the decompressor has to order them via a heap).

However, as impressive as many of these features are, many applications need a text log immediately available, without having to run a decompressor, and I think it would help many applications out there if NanoLog could, instead of running a compressor thread, run a text writer thread, which would dequeue string buffers with log data, order them by time (maybe within some limited time frame window), format log entries as text and write to a text log file.

Along the same lines, NanoLog could also provide a method for applications to call on their own thread, which would do all of the above and return a formatted log line or a log buffer that applications could use any way they see fit, such as dumping them to a file, network, etc.

Infinite Loop in Makefile Dependency Generation

In runtime/GNUmakefile, dependency generation uses the following sed script to generate dependencies for sources used in tests:

sed 's|[a-zA-Z0-9_-]*\.o|$(TEST_BUILD_DIR)/&|' ./.depend >> ./.depend

See

NanoLog/runtime/GNUmakefile

Line 76 in 2a94d70

sed 's|[a-zA-Z0-9_-]*\.o|$(TEST_BUILD_DIR)/&|' ./.depend >> ./.depend

.

The issue with this script is that ./.depend can be written to before the sed script completes, sending the script into an infinite loop. This behavior is being triggered for me in WSL1 Ubuntu 18.04.

I'll follow up with a PR to break the lib and test dependencies in two.

std::string std::string_view support

How much work would it be to add support for std::string and std::string_view and thus avoid the use of strlen() in the hotpath?

Currently using std::string_view is quite problematic as there's no guarantee that it's null terminated so you have to construct a std::string and then call c_str() on it before passing it into the logger. Considering the length is already encoded into the std::strig_view this is very wasteful.

Microsoft Windows Support?

Is Microsoft Windows works with this library? If yes then how?

No CMake integration

Would it be possible for you to add CMake integration for this project? It is one of the most popular build systems in C++ ecosystem and it would increase the ease of use of NanoLog in other projects.

how to use in windows?can someone give guide?

Allocation fails with printing concatenated string / integer data

For either the benchmarking code or the sample application, switching
"NANO_LOG(NOTICE, \"Starting backup replica garbage collector thread\");"
for
NANO_LOG(NOTICE, "A string, pointer, number, and float: '%s', %p, %d, %f", randomString, &randomString, 512, 3.14159);
in the logging loop causes RuntimeLogger::reserveAlloc() to fail.
Platform: sles12sp1, Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz, 125GB of RAM.
NanoLog will write a random (usually between 50 and 200 MB) amount of data before the thread crashes and NanoLog is stuck waiting for more data to compress.