abumq/residue

Real-time centralized logging server ⚡

Note

☝️ This is not a true representation of most used languages, as I have over 30 private repositories in my personal Github account and over 50 private repositories spread across 3 personally owned organizations. These repositories use mixture of C++, BASH, SQL, Javascript (for Vanilla JS, React and Node.js) and Python (for ML/AI, Jupyter notebooks, APIs and scripts)

Sponsorship

If my work benefits you or your organization in any way, or you want to support me by resuming work on my dream project, please consider supporting me via Github sponsorship program.

residue's Issues

Clients added dynamically need processor

Since 1.6.0 (the custom queue processor feature) we didn't add custom processors for dynamically added clients

Separate log dispatchers and logging queue by clients

This is subject to issue 613 on Easylogging++. That needs to be fixed before this should be worked on otherwise we may run in to race condition

This is an improvement to prevent one client blocking other client's logs in case of “rush hour”

We will have separate queue for unknown clients and one queue for each known client initialized at the startup

Ensure reloading configs reconfigure loggers

We need to ensure reloading config reconfigures existing loggers

Long pending requests fail after dead client's key is reset

Details

If we have long running log queue and following things happen all at once:

• process hasn't finished before client is dead
• client has short life
• no request made to client (touch) in the mean time
• after client is dead and before integrity task is executed, request is made (this will reset the key)
• there are remaining requests in backlog

all the subsequent request in next backlog process will fail

This is very rare situation and we only found this out after crazy load testing and really hammering the server.

Remove log extension

We can remove log-extension from residue as we are continuing this idea from resitail hooks

However, we will keep the concept of extension for other types of events:

• Log rotation
• Log archiving
• Handle crashes

Demo docs

Details

Currently demo docs are not valid as we have shut down demo server.

Expected

Change the docs so user can create their own fully function demo server (both closed and open server)

also clean /demo directory to properly demonstrate the usage of the server.

Remove plain log request support

This is a security concern + extra instructions for potentially unused feature

We will remove it in 1.5.0

Reload logger configuration

Details

Provide CLI and admin request to be able to reload the registered logger configuration

Crash when invalid specific request sent

Details

Request will crash the server

printf "{\"type\":[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[,\"rsa_public_key\":\"`cat client-256-public.pem | ripe -e --base64`\"}\r\n\r\n" | nc `cat host` 8777

Affected Version

v1.4.2

Remove token server

We will remove token server as we do not need it after we removed plain log request support

This decision is subject to change. We need to further think on whether we really want to do it or it's necessary.

[This will break existing client libraries as we will not have token port available]

This has following impacts

• Remove requires_timestamp as it will always be required
• Change LogRequest::validateTimestamp to check for time stamps
• token_age, max_token_age, access codes concepts, requires_token can all be removed from code base

NOTE: We need to consider different timezones as requiring timestamps can be an issue in such cases

Client integrity task can remove pending dead client

Details

If log processing is running behind and one of the clients are past their age and client integrity task runs in the mean time, it will remove the client from registry. This will cause the log to fail with invalid message [log-request-handler.cc:107] Failed: Client not connected yet

Currently the situation is handled and client is temporarily brought alive if client is DEAD but still registered.

Expected

Log processing should be successful for such dead client because it passed the original validation (at time of connection and token retrieval)

Affected Version

v1.x.x

Race condition causing deadlock when creating log file

When log rotation truncates the file and file is being created. Last log message seen is

22:51:56,265 [AdminHandler] [INFO] FSR: Prepending 'level' format specifier in filename for logger [sample-app] as we have multiple filenames for levels
22:51:56,265 [AdminHandler] [INFO] FSR: [/tmp/logs/sample-app.log] => [/tmp/logs/backups/sample-app/] as [global-2018.log]
22:51:56,265 [AdminHandler] [INFO] FSR: Result: [/tmp/logs/backups/sample-app/] [2018.tar.gz] with [3] items
22:51:56,265 [AdminHandler] [vDETAILS] [log-rotator.cc:272] Ignoring rotating empty file /tmp/logs/sample-app-verbose.log
22:51:56,265 [AdminHandler] [vDETAILS] [log-rotator.cc:272] Ignoring rotating empty file /tmp/logs/sample-app-debug.log
22:51:56,265 [AdminHandler] [vDETAILS] [log-rotator.cc:261] Rotating [/tmp/logs/sample-app.log] => [/tmp/logs/backups/sample-app/global-2018.log] (1.2GB)
22:51:56,265 [LogDispatcher] [vDEBUG] [log-request-handler.cc:185] Force check: 0, clientRef: 0x7000098bddf0, *clientRef: muflihun00102030, bypassChecks: 1
22:51:56,265 [LogDispatcher] [ERROR] File not found [/tmp/logs/sample-app.log] [Logger: sample-app]. Creating...
22:51:56,265 [LogDispatcher] [INFO] Accessing file...

The next line is updating permission but there is no lock required from this point onwards so not sure why is this happening

Log rotator task - daily, weekly, monthly and yearly schedule

Details

Log rotator schedules from when server is started. i.e, if server started at 9:36am the next daily rotation will rotate at 10:00 on that day (it should be next day instead) and subsequent rotations will also be 10:00 each day

This is not expected behaviour as user may expect daily logs to be only archived the next day and not on the next lowest denominator run (i.e, HOURLY)

The hourly rotation rounds off to next 00 minutes instead of the start of server which is a plus and makes this fix easier ✅

Expected

In above scenario the daily rotation should be at next nearest 00:00 and then continue on wards on this time

Same thing should happen for weekly, monthly and yearly tasks

Affected Version

v1.2.3 (all previous versions)

Completely remove old JSON library

JSON for Modern C++ is very good library and we love it. but it has features that we do not need and we needed very basic JSON implementation. We improved compiled time with this

See #81
See #74

We have tested new library and it works beautifully to suit our needs. We will remove old library's code from our codebase

Bulk items do not have correct translations for unsigned long

Details

std::string requestItemStr(d.dump());

causes ints to translate as:

Affected Version

v1.4.3

Deadlock detected while dispatching log

19:03:02,606 [LogHandler] [vTRACE] [session.cc:86] Adding bytes
19:03:02,608 [LogDispatcher] [vDEBUG] [log-request-handler.cc:88] Pausing schedule for client integrity
19:03:02,609 [LogHandler] [vTRACE] [session.cc:75] Received: 522 bytes
19:03:02,609 [LogHandler] [vTRACE] [session.cc:102] Read bytes: [..snip] [size: 522]
19:03:02,609 [LogHandler] [vDEBUG] [session.cc:171] Sending {r:0}


19:03:02,609 [LogHandler] [vTRACE] [session.cc:86] Adding bytes
19:03:02,609 [LogDispatcher] [vCRAZY] [log-request-handler.cc:100] -----============= [ BEGIN ] =============-----
19:03:02,610 [LogDispatcher] [vDEBUG] [request-handler.h:144] Raw request: [..snip]
19:03:02,610 [LogDispatcher] [vDEBUG] [request-handler.cc:58] Client: [..snip]
19:03:02,610 [LogDispatcher] [vCRAZY] [request-handler.cc:59] IV: [..snip]
19:03:02,610 [LogDispatcher] [vCRAZY] [request-handler.cc:66] Data (base64): [..snip]
19:03:02,610 [LogDispatcher] [vCRAZY] [request-handler.cc:69] Ripe command: [..snip]
19:03:02,610 [LogDispatcher] [vCRAZY] [request-handler.cc:76] Plain request: [..snip]
19:03:02,610 [LogDispatcher] [vTRACE] [request-handler.h:148] Decryption finished (b64): [..snip]
19:03:02,610 [LogDispatcher] [vTRACE] [request-handler.h:159] Decompressing: [..snip]
19:03:02,611 [LogDispatcher] [vTRACE] [request-handler.h:170] Decompression finished (raw): [{... (removed)}]
19:03:02,612 [LogDispatcher] [vDEBUG] [log-request-handler.cc:115] Is bulk? true
19:03:02,612 [LogDispatcher] [vDEBUG] [log-request-handler.cc:124] Request client: 0x10065cfa8
19:03:02,612 [LogHandler] [vTRACE] [session.cc:75] Received: 522 bytes
19:03:02,612 [LogDispatcher] [vDEBUG] [log-request-handler.cc:206] Force check: true, clientRef: 0x70000f9729a0, *clientRef: muflihun00102030, bypassChecks: false
19:03:02,612 [LogHandler] [vTRACE] [session.cc:102] Read bytes: 29277885c8e3830c20fb91fab830194b:muflihun00102030:zaAqRKANlz9SElDUXg6ePbeykddpAyJjxC3TEL/yutU5HIryqLpN1xmsEncDHizpniIeaCIY7kfUjuzTaXwYJC0dxHxlssq5Lm7PzgPCyyxa7POh0quW1zooTSlBMvUiuCWamAdfzz3WPPjjbMSyvqtgMmbFIhs4qMMMt3kKZbli8kk9Hmv5XV8Ltz+/tLX4ppiOn6qSFh/pIQ0LNaBVK0wHXLfGO+AN58vc1D50JT8vA00XUjkzNNmeasekIntZDoRut/WhgqVijoUsT27sY9WzrsdaLuIJIAUhtewqvAGgyeplI+lucgHWtWKVR+HQQ887giQwW6+80FrU0yIZ/yEhDnFbCul3Zc9L3vuuBHvrAgjYvDPkA4MZPwBkFWSh6anUbJ5zsNgEnMP2tdDI+ZLnSVfvQ+ZNEPeVXYE15la/XVPoe0/b/2MqY3HPUrYEtTJONqT/XrZKl1HXbqxp4w== [size: 522]
19:03:02,613 [LogDispatcher] [vDEBUG] [client.cc:65] Checking token 37953058 (client [muflihun00102030])

Unable to parse strings with double quotes

Details

JSON parser unable to parse string with double quotes. This must be fixed as messages may contain double quotes

Affected Version

v1.4.3

Adding to log queue needs re-thinking

We need to re-think the way when we respond back to client for the log request. We hold the shared pointer to the session which causes client to potentially wait until the log is processed (unless _close flag is on)

Crash noticed during load testing

22:30:51,352 [LogDispatcher] [FATAL] CRASH HANDLED; Application has crashed due to [SIGSEGV] signal
    ======= Backtrace: =========
    [1] 2   residue                             0x000000010c5c690b _Z23generalTerminateHandlerib + 267
    [2] 3   libsystem_platform.dylib            0x00007fff6865ef5a _sigtramp + 26
    [3] 4   ???                                 0x0000000000000002 0x0 + 2
    [4] 5   residue                             0x000000010c66d08d _ZNSt3__16__treeINS_12__value_typeINS_12basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEEEN8nlohmann10basic_jsonINS_3mapENS_6vectorES7_bxydS5_NS8_14adl_serializerEEEEENS_19__map_value_compareIS7_SE_NS_4lessIS7_EELb1EEENS5_ISE_EEE7destroyEPNS_11__tree_nodeISE_PvEE + 29
    [5] 6   residue                             0x000000010c66d08d _ZNSt3__16__treeINS_12__value_typeINS_12basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEEEN8nlohmann10basic_jsonINS_3mapENS_6vectorES7_bxydS5_NS8_14adl_serializerEEEEENS_19__map_value_compareIS7_SE_NS_4lessIS7_EELb1EEENS5_ISE_EEE7destroyEPNS_11__tree_nodeISE_PvEE + 29
    [6] 7   residue                             0x000000010c66d08d _ZNSt3__16__treeINS_12__value_typeINS_12basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEEEN8nlohmann10basic_jsonINS_3mapENS_6vectorES7_bxydS5_NS8_14adl_serializerEEEEENS_19__map_value_compareIS7_SE_NS_4lessIS7_EELb1EEENS5_ISE_EEE7destroyEPNS_11__tree_nodeISE_PvEE + 29
    [7] 8   residue                             0x000000010c66d099 _ZNSt3__16__treeINS_12__value_typeINS_12basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEEEN8nlohmann10basic_jsonINS_3mapENS_6vectorES7_bxydS5_NS8_14adl_serializerEEEEENS_19__map_value_compareIS7_SE_NS_4lessIS7_EELb1EEENS5_ISE_EEE7destroyEPNS_11__tree_nodeISE_PvEE + 41
    [8] 9   residue                             0x000000010c66d099 _ZNSt3__16__treeINS_12__value_typeINS_12basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEEEN8nlohmann10basic_jsonINS_3mapENS_6vectorES7_bxydS5_NS8_14adl_serializerEEEEENS_19__map_value_compareIS7_SE_NS_4lessIS7_EELb1EEENS5_ISE_EEE7destroyEPNS_11__tree_nodeISE_PvEE + 41
    [9] 10  residue                             0x000000010c66d08d _ZNSt3__16__treeINS_12__value_typeINS_12basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEEEN8nlohmann10basic_jsonINS_3mapENS_6vectorES7_bxydS5_NS8_14adl_serializerEEEEENS_19__map_value_compareIS7_SE_NS_4lessIS7_EELb1EEENS5_ISE_EEE7destroyEPNS_11__tree_nodeISE_PvEE + 29
    [10] 11  residue                             0x000000010c66cf78 _ZN8nlohmann10basic_jsonINSt3__13mapENS1_6vectorENS1_12basic_stringIcNS1_11char_traitsIcEENS1_9allocatorIcEEEEbxydS7_NS_14adl_serializerEED2Ev + 56
    [11] 12  residue                             0x000000010c66cfad _ZN8nlohmann10basic_jsonINSt3__13mapENS1_6vectorENS1_12basic_stringIcNS1_11char_traitsIcEENS1_9allocatorIcEEEEbxydS7_NS_14adl_serializerEED2Ev + 109
    [12] 13  residue                             0x000000010c61d833 _ZN7residue7RequestD2Ev + 83
    [13] 14  residue                             0x000000010c619581 _ZN7residue17LogRequestHandler19processRequestQueueEv + 4449
    [14] 15  residue                             0x000000010c61d8e8 _ZNSt3__114__thread_proxyINS_5tupleIJNS_10unique_ptrINS_15__thread_structENS_14default_deleteIS3_EEEEZN7residue17LogRequestHandler5startEvE3$_0EEEEEPvSB_ + 152
    [15] 16  libsystem_pthread.dylib             0x00007fff686686c1 _pthread_body + 340
    [16] 17  libsystem_pthread.dylib             0x00007fff6866856d _pthread_body + 0
    [17] 18  libsystem_pthread.dylib             0x00007fff68667c5d thread_start + 13

Cross macOS (High Sierra) not able to start

Compiled on one macOS and ran on another caused this failure to startup

Application abnormally terminated.
Please report it to us on https://github.com/muflihun/residue/issues
2018-03-02 21:11:57,264 FATAL [residue] CRASH HANDLED; Application has crashed due to [SIGILL] signal
    ======= Backtrace: =========
    [1] 2   residue                             0x000000010c9ef64b residue + 9803
    [2] 3   libsystem_platform.dylib            0x00007fff73d1af5a _sigtramp + 26
    [3] 4   residue                             0x000000010cc27935 _ZTIN8CryptoPP31IteratedHashWithStaticTransformIjNS_10EnumToTypeINS_9ByteOrderELi1EEELj64ELj32ENS_6SHA224ELj28ELb1EEE + 240517
    [4] 5   residue                             0x000000010ca9ec4b _ZN4asio6detail28reactive_socket_send_op_baseINS_15const_buffers_1EE10do_performEPNS0_10reactor_opE + 199643
    [5] 6   residue                             0x000000010caab090 _ZN4asio6detail28reactive_socket_send_op_baseINS_15const_buffers_1EE10do_performEPNS0_10reactor_opE + 249888
    [6] 7   residue                             0x000000010caab1c0 _ZN4asio6detail28reactive_socket_send_op_baseINS_15const_buffers_1EE10do_performEPNS0_10reactor_opE + 250192
    [7] 8   residue                             0x000000010caabe27 _ZN4asio6detail28reactive_socket_send_op_baseINS_15const_buffers_1EE10do_performEPNS0_10reactor_opE + 253367
    [8] 9   residue                             0x000000010caaf26c _ZN4asio6detail28reactive_socket_send_op_baseINS_15const_buffers_1EE10do_performEPNS0_10reactor_opE + 266748
    [9] 10  residue                             0x000000010caaf390 _ZN4asio6detail28reactive_socket_send_op_baseINS_15const_buffers_1EE10do_performEPNS0_10reactor_opE + 267040
    [10] 11  residue                             0x000000010cac2d0a _ZNK8CryptoPP12AbstractRingINS_7IntegerEE12ExponentiateERKS1_S4_ + 58
    [11] 12  residue                             0x000000010caacf09 _ZN4asio6detail28reactive_socket_send_op_baseINS_15const_buffers_1EE10do_performEPNS0_10reactor_opE + 257689
    [12] 13  residue                             0x000000010cb59ccd _ZN8CryptoPP16IteratedHashBaseIjNS_25MessageAuthenticationCodeEE18HashMultipleBlocksEPKjm + 237453
    [13] 14  residue                             0x000000010ca31612 residue + 280082
    [14] 15  residue                             0x000000010cb53192 _ZN8CryptoPP16IteratedHashBaseIjNS_25MessageAuthenticationCodeEE18HashMultipleBlocksEPKjm + 210002
    [15] 16  residue                             0x000000010ca92e18 _ZN4asio6detail28reactive_socket_send_op_baseINS_15const_buffers_1EE10do_performEPNS0_10reactor_opE + 150952
    [16] 17  residue                             0x000000010cb06814 _ZNK8CryptoPP4OAEPINS_4SHA1ENS_10P1363_MGF1EE6NewMGFEv + 135508
    [17] 18  residue                             0x000000010ca918e3 _ZN4asio6detail28reactive_socket_send_op_baseINS_15const_buffers_1EE10do_performEPNS0_10reactor_opE + 145523
    [18] 19  residue                             0x000000010ca91a3b _ZN4asio6detail28reactive_socket_send_op_baseINS_15const_buffers_1EE10do_performEPNS0_10reactor_opE + 145867
    [19] 20  residue                             0x000000010ca2fe8a residue + 274058
    [20] 21  residue                             0x000000010ca28c07 residue + 244743
    [21] 22  residue                             0x000000010ca6643b residue + 496699
    [22] 23  residue                             0x000000010ca50c63 residue + 408675
    [23] 24  residue                             0x000000010ca4dd0d residue + 396557
    [24] 25  residue                             0x000000010ca4d2c9 residue + 393929
    [25] 26  residue                             0x000000010c9f0953 residue + 14675
    [26] 27  libdyld.dylib                       0x00007fff73a99115 start + 1
    [27] 28  ???                                 0x0000000000000003 0x0 + 3

Adding dynamically added client does not save this client

Because configuration::save uses original json

Crash detected when writing to session

Details

Crash detected - possibly ASIO bug but need to be able to reproduce

Here is stacktrace and logs before crash

2018-02-15 12:00:01,841 [LogHandler] [residue] [vDEBUG] [session.cc:54 (residue::Session::~Session())] End session LwaRHNfcSvOmfdHIl
2018-02-15 12:00:01,841 [LogHandler] [residue] [vTRACE] [session.cc:102 (void residue::Session::sendToHandler(std::string&&))] Read bytes: 2f3056d19c96efdcee05f84b17aafca1:muflihun:XxOIBAjAvSHVGuzQmvewGc584c5Yt8tE0ZwFgP1tj5IBc25DeHxZmkMq6rFZMrAX8WICvBko2e2Rs7qAvC6hZI9dCSC6aKtoqz8NXNViJiWRsy2UyWBOT4a+nF3AH3eS3XhBm8ut63VyWig2kwbj3/NhGj0pmrLX5HlZl54A6rFMmFuRQi7X7+n+fiJhu1YE7L1i6QPL7rXWIa3+BRXeHzuJ9ASCsrSdMDs2YoZVB/mv91nsDzvN7gqPN7M4zYKFDbT/DtPoey0jejnsjvjjhtxHOCoYp79GQIBXim7BJBtzTyK35FHotjmsEYLCvVELZ+Pv9eLaD/TyXvjpv+ErWRHLn3OysPwe8ykV/tVM85c= [size: 406]
2018-02-15 12:00:01,842 [LogHandler] [residue] [FATAL] CRASH HANDLED; Application has crashed due to [SIGSEGV] signal
    ======= Backtrace: =========
    [1] /root/residue-1.4.2-linux-debug-x86_64/residue:generalTerminateHandler(int, bool)+0x23 [0x4eaae3]
    [2] /lib64/libc.so.6+0x347c0 [0x7f21e62057c0]
    [3] /root/residue-1.4.2-linux-debug-x86_64/residue:residue::Session::write(char const*, unsigned long)+0x29 [0x5834f9]
    [4] /root/residue-1.4.2-linux-debug-x86_64/residue:residue::Session::writeStatusCode(residue::Response::StatusCode const&)+0x473 [0x5850d3]
    [5] /root/residue-1.4.2-linux-debug-x86_64/residue:residue::LogRequestHandler::handle(residue::RawRequest&&)+0x32 [0x5405e2]
    [6] /root/residue-1.4.2-linux-debug-x86_64/residue:residue::Session::sendToHandler(std::string&&)+0x737 [0x57f287]
    [7] /root/residue-1.4.2-linux-debug-x86_64/residue() [0x58148d]
    [8] /root/residue-1.4.2-linux-debug-x86_64/residue() [0x582ec0]
    [9] /root/residue-1.4.2-linux-debug-x86_64/residue:asio::detail::epoll_reactor::descriptor_state::do_complete(asio::detail::task_io_service*, asio::detail::task_io_service_operation*, std::error_code const&, unsigned long)+0x13c [0x57b36c]
    [10] /root/residue-1.4.2-linux-debug-x86_64/residue:asio::detail::task_io_service::run(std::error_code&)+0x411 [0x4efa61]
    [11] /root/residue-1.4.2-linux-debug-x86_64/residue() [0x4edc22]
    [12] /lib64/libstdc++.so.6+0xbb9b0 [0x7f21e6b6e9b0]
    [13] /lib64/libpthread.so.0+0x75ca [0x7f21e70585ca]
    [14] /lib64/libc.so.6:clone+0x6d [0x7f21e62d40cd]

Affected Version

v1.4.2