ess-dmsc/graylog-logger

A simple C++ library for writing log messages to console, file and Graylog server

Graylog logger

This is a simple logging library which can be used to send log messages to a Graylog server. This is done by creating messages in the GELF format and sending them to a Graylog server via TCP. For testing purposes a Vagrant machine running Graylog can be used. A simple Vagrantfile for creating this set-up can be found here. The argument for creating yet another logging library instead of writing a plugin/sink/handler for an already existing one is that a relatively light weight solution was desired. The library has functionality for writing log messages to console and file as well. By default the library will only write log messages to console.

The repository is split into four parts:

• The logging library.
• Unit tests of the logging library which are completely self contained (i.e. does not require a Graylog server).
• A simple console application which uses the logging library.
• Some benchmarking code which is used for profiling and optimising the code as well as test for performance regression.

A sister project to this library is the Python logging handler GraylogHandler.

• Further documentation can be found here.

Getting Started

These instructions will get you a copy of the project up and running on your local machine for development and testing purposes. See deployment for notes on how to deploy the project on a live system.

Prerequisites

The logging library has the following (requried) external dependencies:

• moodycamel::ConcurrentQueue For message passing to and within the logging library.
• JSONForModernCPP For generating graylog GELF messages.
• ASIO For networking.

This library can also make use of the following (optional) dependencies:

• libfmt For doing fast (multi-threaded) string formatting.
• GTest For unit testing.
• google-benchmark For benchmarking.

You will also need CMake (version ≥ 3.9) to build the project. The project makes use of library and language features provided by C++14. It might be possible to link to the library using compilers that only supports C++11 though this has not been tested.

Due to the use of ASIO, the library should compile on most *nix systems and Windows 10 with no issues though only limited testing has been done.

Installing

There are two methods for building this library. They are described below.

Using conan

When using conan to provide the dependencies, all the optional and required dependencies are provided automatically. To checkout and build the library, run the following commands:

git clone https://github.com/ess-dmsc/graylog-logger.git
cd graylog-logger
mkdir build
cd build
conan install .. --build=outdated
cmake ..
make install

Making a conan package

If you are creating a conan package of this library you should disable CMake from running Conan by providing the following argument to CMake: -DCONAN=DISABLE. Furthermore, you must also use the cmake_find_package generator for CMake to be able to find the dependencies.

System installed dependencies

If using conan is not an option, it is possible to build the library using system installed dependencies. This requires a bit more work though and might not be as reliable.

Run the following commands:

git clone https://github.com/ess-dmsc/graylog-logger.git
cd graylog-logger
mkdir build
cd build
cmake .. -DCONAN=DISABLE -DCMAKE_PREFIX_PATH=/path/to/dir/containing/the/concurrentqueue/directory/
make install

Documentation

The code has some documentation. To generate it, run doxygen in the root of the repository i.e.:

cd graylog-logger
doxygen

Running the tests

If compiled, the unit tests are run by running the unit_tests application in the unit_tests directory of the builddirectory.

Deployment

Examples illustrating how the library can be used can be found in the examples.md file.

Built With

• CMAKE - Cross platform makefile generation
• Conan - Package manager for C++

Contributing

Get in contact by creating an issue. I will be happy to assist you making the changes that you want.

Versioning

We use SemVer for versioning. For the versions available, see the releases on github.

See description of changes

Authors

• Jonas Nilsson - Initial work - SkyToGround
• Afonso Mukai - Provided much assistance in setting up the continous integration system.
• Matthew Jones - Improved the CMake code and suggested improvements to the interface.

See also the list of contributors to this project.

License

This project is licensed under the BSD-2 License - see the LICENSE.md file for details

graylog-logger's Issues

Rotate log files

Support MSVC

Issues with new Fmt support

Perhaps I'm misunderstanding how this intended to be built but I found that when I build the latest version with conan it doesn't find the fmt library even when I have the fmt package in my conan dependencies.

If I add this to cmake/Findfmt.cmake then it works: https://github.com/ceph/ceph/blob/master/cmake/modules/Findfmt.cmake
Unfortunately that repo is LGPL which is possibly incompatible with your license but we can probably write something from scratch or source from another project to get the same result.

edit: I also get long compiler errors from the use of minimal::apply in FmtMsg using gcc 6.5.0 with -std=c++1z which I would have expected to work if only C++14 features are required.

I'm really more of a C programmer so once things get heavy with templates I'm often a bit lost. But is there a reason to use minimal::apply? My best guess was you had trouble capturing args so ended up using a tuple. But I found that this compiles under --std=gnu++14 and seems to run correctly. Is what I've done here OK or are there cases it wouldn't work?

  template <typename... Args>
  void fmt_log(const Severity Level, std::string Format, Args... args) {
    if (int(Level) > int(MinSeverity)) {
      return;
    }
    auto ThreadId = std::this_thread::get_id();
    Executor.SendWork([=]() {
      LogMessage cMsg(BaseMsg);
      cMsg.SeverityLevel = Level;
      cMsg.Timestamp = std::chrono::system_clock::now();
      try {
          cMsg.MessageString = fmt::format(Format, args...);
      } catch (fmt::format_error &e) {
          cMsg.SeverityLevel = Log::Severity::Error;
          cMsg.MessageString = fmt::format("graylog-logger internal error. Unable to format "
                              "the string \"{}\". The error was: \"{}\".",
                              Format, e.what());
      }
      std::ostringstream ss;
      ss << ThreadId;
      cMsg.ThreadId = ss.str();
      for (auto &ptr : Handlers) {
        ptr->addMessage(cMsg);
      }
    });
  }

Happy to prepare a PR if there's a solution that works for everyone here.

Implement as a spdlog plugin?

There is currently some messy code in the file writer and forwarder for logging to file, to console or to graylog. It seems reasonable to support different logging methods as institutions other than ESS may not want to be bound to Graylog.
One nice way of cleaning up that code would be to use https://github.com/gabime/spdlog and incorporate graylog-logger as a plugin. Any thoughts on this @SkyToGround?

"Informational" sounds awkward

It's probably better to just say "Information" or even "Info"

ASIO need not be exposed in public headers

PIMPL could be used in GraylogConnection.hpp to avoid needing to expose asio in the library headers.

[feature request] Ability to flush logs

I was just debugging a startup issue in our program where it locks up. It was hard to figure out from the logs (configured to output to both Console and GELF) where it got to because several messages remain in the buffer and are never flushed out.

It would be nice if I could manually flush the logs out, or even better if the logging thread flushed them after a certain timeout. I would be happy to prepare a PR if you are interested but I would probably need a little guidance around the best way to add it to the current architecture.

Resource deadlock avoided (terminate called after throwing an instance of 'std::system_error')

When quitting my application (CTRL+C), I am getting the following error.

terminate called after throwing an instance of 'std::system_error'
  what():  Resource deadlock avoided
Abort (core dumped)

It occurs in the GraylogConnection.cpp file on line 64, at connectionThread.join().

void GraylogConnection::EndThread() {
    closeThread = true;
    connectionThread.join();
    if (NULL != conAddresses) {
        freeaddrinfo(conAddresses);
        conAddresses = NULL;
    }
}

In my code, I am opening the Graylog connection like this;

void GrayLogging::open()
{
    handler = new GraylogInterface("myhost"), 11111);

    Log::RemoveAllHandlers();
    Log::AddLogHandler(handler);
    Log::SetMinimumSeverity(Severity::Informational);
}

And when receiving a quit signal, I am deleting the handler, which calls the destructor.

void GrayLogging::close()
{
    delete handler;
}

Do you have any idea how to fix this?

Better error handling for file writing

tensorflow-terminate called after throwing an instance of 'std::system_error'