Lightweight open-source microservice logging framework for the KumuluzEE framework
KumuluzEE Logs is a lightweight open-source logging framework specifically designed for logging microservices.
It provides easy and efficient logging of common log events, such as logging method entries and exits, logging external resource invocations and other events. It also provides automated logging of parameters and performance metrics.
KumuluzEE Logs has been designed to simplify logging for the developer. It introduces a @Log
annotation, which can be used on a class or on a method to enable legging the method entry, parameters, method exit and optionally include performance metrics.
In addition, KumuluzEE Logs also support logging with explicit commands. It provides logging methods for the developer, which can be used in the code directly.
KumuluzEE Logs acts as a façade and provides a simple, common interface with the objective to abstract the underlying logging framework. This makes the logging process easier for the developer, standardizes how the logging is performed, and makes the code independent of the underlying logging framework.
KumuluzEE Logs is designed to support different logging frameworks. Currently, KumuluzEE Logs provides support for Log4J2, java.util.logging (JUL) and Fluentd. In the future, other logging frameworks will be supported too (contributions are welcome).
To address the needs specific to logging microservices, KumuluzEE Logs can be easily configured to collect distributed logs into a centralized log management system, such as Elastic Stack, Graylog, Splunk, etc. Furthermore, KumuluzEE Logs provides support for Apache Kafka and other approaches.
KumuluzEE defines interfaces for common logging features. Therefore, to use the logging you need to include a dependency to implementation library. Currently, Log4j2, JUL and Fluentd are supported. Log4j2 is more appropriate for complex and enterprise grade logging scenarios, while JUL is adequate for simpler logging. Fluentd enables you to unify data collection and consumption for a better use and understanding of data.
To use KumuluzEE Logs with Log4j2, use the following dependency:
<dependency>
<artifactId>kumuluzee-logs-log4j2</artifactId>
<groupId>com.kumuluz.ee.logs</groupId>
<version>${kumuluzee-logs.version}</version>
</dependency>
To use KumuluzEE Logs with JUL, use the following dependency:
<dependency>
<artifactId>kumuluzee-logs-jul</artifactId>
<groupId>com.kumuluz.ee.logs</groupId>
<version>${kumuluzee-logs.version}</version>
</dependency>
To use KumuluzEE Logs with Fluentd, use the following dependency:
<dependency>
<artifactId>kumuluzee-logs-fluentd</artifactId>
<groupId>com.kumuluz.ee.logs</groupId>
<version>${kumuluzee-logs.version}</version>
</dependency>
You can use one dependency only. You cannot use both dependencies at the same time.
To use the developer logging functionality get a new Logger
instance by using LogManager
:
private static final Logger LOG = LogManager.getLogger(CustomerResourceSample.class.getName());
Logger defines multiple overloaded methods for each logging level (error, warning, info, debug, trace), for example:
LOG.trace("Trace log with String only");
LOG.info("Info log with parameter: {}", someVariable);
LOG.error("Error with exception log", exception);
LOG.error(exception); //exception only
Additional common logging is available through LogCommons
interface, method entry and method exit logging can be used by using @Log
annotation used at class level or method level or by invoking the methods manually. Resource logging can be used only by manual method invocation.
Example for using @Log
at Class level:
@Log
public class SomeClass {
//Implementation
}
You can define additional attributes in @Log
annotation for monitoring method execution duration and disabling method invocation details. Observe three different possibilities for configuring method entry and exit logging:
@Log
@Log(LogParams.METRICS)
@Log(value = LogParams.METRICS, methodCall = false)
Additional common logging is available through LogCommons
interface, method entry and method exit logging can be used by using @Log
annotation used at class level or method level or by invoking the methods manually. Resource logging can be used only by manual method invocation.
Example for using @Log
at Class level:
@Path("customers")
@Log
public class CustomerResource {
//Implementation
}
You can define additional attributes in @Log
annotation for monitoring method execution duration and disabling method invocation details. Observe three different possibilities for configuring method entry and exit logging:
@Log
@Log(LogParams.METRICS)
@Log(value = LogParams.METRICS, methodCall = false)
If you use javax.ws.rs.ext.ExceptionMapper
to map exceptions to error responses, @Log
won't log the method exit when exception is thrown because it is an interceptor and the method exit never happens. If you want to be absolutely sure to log every single API resource entry and exit, use javax.servlet.Filter
or javax.ws.rs.container.ContainerResponseFilter
in addition to or instead of @Log
.
Additional functionality of LogCommons
implementation is the ability to log and monitor invocations of external resources, for example databases and services. Resource monitoring allows you to log resource parameters and performance metrics. This functionality is available only through manual invocation of LogCommons
methods.
Sample code of resource invocation monitoring:
LogResourceContext context = logCommons.logResourceStart(
LogLevel.TRACE,
Marker.DATABASE,
new LogResourceMessage().enableInvoke(invokeMessage).enableMetrics());
//...Read resource by id
logCommons.logResourceEnd(context);
In the sample above, we invoke logResourceStart
method, with parameters:
- LogLevel, which specifies different log levels (TRACE, ERROR, DEBUG...).
- Marker, which is an enum, implementing interface
com.kumuluz.ee.logs.markers.Marker
. You should implement Markers according to your needs. - New
LogResourceMessage
instance, where we set the invocation message (see below) and enable metrics monitoring.
logResourceStart
method returns LogResourceContext
instance, which passes relavant information for logging end of resource invocation.
InvokeMessage
variable is an instsance of com.kumuluz.ee.logs.messages.ResourceInvokeLogMessage
interface and therefore must implement a method which returns a Map (Map<String, String>
) of parameters. For example, the sample of InvocationMessage could be used to populate the Map the in the following way:
InvocationMessage invokeMessage = new InvocationMessage("Invocation of database resource").
addName("User").addParameter("id", id);
Audit is an investigation of an existing system usage and is used to determine trace of system access. It provides evidence if the information systems are safeguarding assets. Audit log is different from system log in a way to provide structured and higher level of information needed to effectively process audit information.
Add audit logs dependency:
<dependency>
<groupId>com.kumuluz.ee.logs</groupId>
<artifactId>kumuluzee-logs-audit</artifactId>
</dependency>
Register AuditLogFilter as JAX-RS component.
@Override
public Set<Class<?>> getClasses() {
Set<Class<?>> classes = new HashSet<>();
classes.add(AuditLogFilter.class);
...
return classes;
}
Due to reduce effort on adding / maintaining annotations Kumuluz Audit generates audit log automatically on all REST resources annotated with @Path.
- audit-action comes from HTTP method and produces values: READ, CREATE, UPDATE, DELETE
- audit-object-type comes from Resource @Path value annotation
- audit-object-id comes from URI or from Location response header for POST requests
Audit log can be overridden with audit annotations, placed on REST resource class or method.
LogAudit annotations can be added to classes or methods. Action name defaults to method name if not specified explicitly in action parameter. AuditObjectParam marks method parameter value to be included in audit log.
@PUT
@LogAudit(name = "user", properties = {
@AuditProperty(property = "action", value = "UPDATE")
})
@Path("/{id}")
public Response updateUser(@AuditObjectParam("id") @PathParam("id") UUID id, @ApiParam(name = "item", required = true) User user) {
//...
}
AuditLogger can be injected into managed CDI bean. Actions or entity changes can be logged with log API.
public class UserResource {
@Inject
private AuditLog auditLog;
public void createUser(Object obj) {
//...
auditLog.log("user", DataAuditAction.CREATE, id, new AuditProperty("customProp", "customValue"));
}
}
Audit log is disabled by default when added as dependency. Audit configuration can be provided at startup or runtime:
kumuluzee:
logs:
audit:
enable: true #false by default
class: com.example.CustomAuditLogger #default is com.kumuluz.ee.logs.audit.loggers.KumuluzAuditLogger
KumuluzEE declares Audit API (available as injection or annotation) and comes with predefined KumuluzAuditLogger
implementation which logs audit lines at INFO level to configured Log implementation (Log4j, JUL, Fluentd...).
KumuluzAuditLogger
implementation logs audit as com.kumuluz.ee.logs.audit.loggers.AuditLogger
category.
KumuluzEE gives you option to provide custom implementation of com.kumuluz.ee.logs.audit.loggers.AuditLogger
via configuration property kumuluzee.logs.audit.class
.
KumuluzEE Logs use the KumuluzEE Config framework to provide configuration of the logging framework. The following options are available:
- Loggers with names and levels
- Config file content
- Config file location
Configuration can be provided at startup or runtime. For runtime configuration, you have to use the KumuluzEE Config project for config servers (etcd or Consul).
Logger levels can be configured individually using the kumuluzee.logs.loggers
config property:
kumuluzee:
logs:
loggers:
- name: com.kumuluz.ee.samples.kumuluzee_logs.CustomerResource
level: TRACE
- name: ''
level: INFO
The root logger can be referenced by providing an empty string ('') or a combination of whitespaces (useful for Consul), which will be trimmed to an empty string.
Instead of using the config file in the file system, which might be inappropriate for microservices, the content of the config file can be provided within the kumuluzee.logs.config-file
config property, as shown below for Log4j2. You can provide config properties for JUL in the same way.
kumuluzee:
logs:
config-file: '<?xml version="1.0" encoding="UTF-8"?>
<Configuration name="customers">
<Appenders>
<Console name="console" target="SYSTEM_OUT">
<PatternLayout pattern="%d %p %marker %m %X %ex %n"/>
</Console>
</Appenders>
<Loggers>
<!-- Default logger -->
<Root level="info">
<AppenderRef ref="console"/>
</Root>
</Loggers>
</Configuration>'
If you prefer to use the config file in the file system, you can specify the config file location using the kumuluzee.logs.config-file-location
config property:
kumuluzee:
logs:
config-file-location: /home/kumuluz/kumuluzee-samples/kumuluzee-logs-log4j2/src/main/resources/log4j2.xml
At startup, if both kumuluzee.logs.config-file
and kumuluzee.logs.config-file-location
are provided the kumuluzee.logs.config-file
will have priority. When configuring with Consul this is not the the case since the last sent value will have priority.
The configuration for the Log4j2 library must be available for the application to load it. Please refer to Log4j2 documentation for rules regarding how the configuration is loaded. Configuration should be in the file named log4j2.xml
, which is located by default in src/main/resources
:
<?xml version="1.0" encoding="UTF-8"?>
<Configuration name="config-name">
<Appenders>
<Console name="console" target="SYSTEM_OUT">
<PatternLayout pattern="%d %p %marker %m %X %ex %n"/>
</Console>
</Appenders>
<Loggers>
<Root level="info">
<AppenderRef ref="console"/>
</Root>
</Loggers>
</Configuration>
The configuration for JUL library will be loaded from the JRE logging.properties file. You can however provide your own logging.properties configuration file and enabling it by providing -Djava.util.logging.config.file
system property. Sample configuration, which should be in a file named logging.properties
and located in src/main/resources
:
# Default global logging level
.level=FINER
# ConsoleHandler definition
handlers=java.util.logging.ConsoleHandler
# ConsoleHandler configuration settings
java.util.logging.ConsoleHandler.level=FINER
java.util.logging.ConsoleHandler.formatter=java.util.logging.SimpleFormatter
The configuration for Fluentd is loaded from the config.yaml
configuration file or other specified
configuration source. Configuration for Fluentd daemon is configured through the daemon configuration file. Each log
message is automatically enriched with the contextual data, if the following configuration values are set: kumuluzee
.name,
kumuluzee.env.name and kumuluzee.version. Sample configuration:
kumuluzee:
name: fluentd-sample
env:
name: dev
version: 1.0.0
logs:
fluentd:
hostname: localhost
port: 24224
We also recommend checking the kumuluzee-logs-fluentd-sample.
Recent changes can be viewed on Github on the Releases Page
See the contributing docs
When submitting an issue, please follow the guidelines.
When submitting a bugfix, write a test that exposes the bug and fails before applying your fix. Submit the test alongside the fix.
When submitting a new feature, add tests that cover the feature.
MIT