abumq / residue Goto Github PK
View Code? Open in Web Editor NEWReal-time centralized logging server ⚡
License: Other
Real-time centralized logging server ⚡
License: Other
Crash detected - possibly ASIO bug but need to be able to reproduce
Here is stacktrace and logs before crash
2018-02-15 12:00:01,841 [LogHandler] [residue] [vDEBUG] [session.cc:54 (residue::Session::~Session())] End session LwaRHNfcSvOmfdHIl
2018-02-15 12:00:01,841 [LogHandler] [residue] [vTRACE] [session.cc:102 (void residue::Session::sendToHandler(std::string&&))] Read bytes: 2f3056d19c96efdcee05f84b17aafca1:muflihun:XxOIBAjAvSHVGuzQmvewGc584c5Yt8tE0ZwFgP1tj5IBc25DeHxZmkMq6rFZMrAX8WICvBko2e2Rs7qAvC6hZI9dCSC6aKtoqz8NXNViJiWRsy2UyWBOT4a+nF3AH3eS3XhBm8ut63VyWig2kwbj3/NhGj0pmrLX5HlZl54A6rFMmFuRQi7X7+n+fiJhu1YE7L1i6QPL7rXWIa3+BRXeHzuJ9ASCsrSdMDs2YoZVB/mv91nsDzvN7gqPN7M4zYKFDbT/DtPoey0jejnsjvjjhtxHOCoYp79GQIBXim7BJBtzTyK35FHotjmsEYLCvVELZ+Pv9eLaD/TyXvjpv+ErWRHLn3OysPwe8ykV/tVM85c= [size: 406]
2018-02-15 12:00:01,842 [LogHandler] [residue] [FATAL] CRASH HANDLED; Application has crashed due to [SIGSEGV] signal
======= Backtrace: =========
[1] /root/residue-1.4.2-linux-debug-x86_64/residue:generalTerminateHandler(int, bool)+0x23 [0x4eaae3]
[2] /lib64/libc.so.6+0x347c0 [0x7f21e62057c0]
[3] /root/residue-1.4.2-linux-debug-x86_64/residue:residue::Session::write(char const*, unsigned long)+0x29 [0x5834f9]
[4] /root/residue-1.4.2-linux-debug-x86_64/residue:residue::Session::writeStatusCode(residue::Response::StatusCode const&)+0x473 [0x5850d3]
[5] /root/residue-1.4.2-linux-debug-x86_64/residue:residue::LogRequestHandler::handle(residue::RawRequest&&)+0x32 [0x5405e2]
[6] /root/residue-1.4.2-linux-debug-x86_64/residue:residue::Session::sendToHandler(std::string&&)+0x737 [0x57f287]
[7] /root/residue-1.4.2-linux-debug-x86_64/residue() [0x58148d]
[8] /root/residue-1.4.2-linux-debug-x86_64/residue() [0x582ec0]
[9] /root/residue-1.4.2-linux-debug-x86_64/residue:asio::detail::epoll_reactor::descriptor_state::do_complete(asio::detail::task_io_service*, asio::detail::task_io_service_operation*, std::error_code const&, unsigned long)+0x13c [0x57b36c]
[10] /root/residue-1.4.2-linux-debug-x86_64/residue:asio::detail::task_io_service::run(std::error_code&)+0x411 [0x4efa61]
[11] /root/residue-1.4.2-linux-debug-x86_64/residue() [0x4edc22]
[12] /lib64/libstdc++.so.6+0xbb9b0 [0x7f21e6b6e9b0]
[13] /lib64/libpthread.so.0+0x75ca [0x7f21e70585ca]
[14] /lib64/libc.so.6:clone+0x6d [0x7f21e62d40cd]
v1.4.2
Compiled on one macOS and ran on another caused this failure to startup
Application abnormally terminated.
Please report it to us on https://github.com/muflihun/residue/issues
2018-03-02 21:11:57,264 FATAL [residue] CRASH HANDLED; Application has crashed due to [SIGILL] signal
======= Backtrace: =========
[1] 2 residue 0x000000010c9ef64b residue + 9803
[2] 3 libsystem_platform.dylib 0x00007fff73d1af5a _sigtramp + 26
[3] 4 residue 0x000000010cc27935 _ZTIN8CryptoPP31IteratedHashWithStaticTransformIjNS_10EnumToTypeINS_9ByteOrderELi1EEELj64ELj32ENS_6SHA224ELj28ELb1EEE + 240517
[4] 5 residue 0x000000010ca9ec4b _ZN4asio6detail28reactive_socket_send_op_baseINS_15const_buffers_1EE10do_performEPNS0_10reactor_opE + 199643
[5] 6 residue 0x000000010caab090 _ZN4asio6detail28reactive_socket_send_op_baseINS_15const_buffers_1EE10do_performEPNS0_10reactor_opE + 249888
[6] 7 residue 0x000000010caab1c0 _ZN4asio6detail28reactive_socket_send_op_baseINS_15const_buffers_1EE10do_performEPNS0_10reactor_opE + 250192
[7] 8 residue 0x000000010caabe27 _ZN4asio6detail28reactive_socket_send_op_baseINS_15const_buffers_1EE10do_performEPNS0_10reactor_opE + 253367
[8] 9 residue 0x000000010caaf26c _ZN4asio6detail28reactive_socket_send_op_baseINS_15const_buffers_1EE10do_performEPNS0_10reactor_opE + 266748
[9] 10 residue 0x000000010caaf390 _ZN4asio6detail28reactive_socket_send_op_baseINS_15const_buffers_1EE10do_performEPNS0_10reactor_opE + 267040
[10] 11 residue 0x000000010cac2d0a _ZNK8CryptoPP12AbstractRingINS_7IntegerEE12ExponentiateERKS1_S4_ + 58
[11] 12 residue 0x000000010caacf09 _ZN4asio6detail28reactive_socket_send_op_baseINS_15const_buffers_1EE10do_performEPNS0_10reactor_opE + 257689
[12] 13 residue 0x000000010cb59ccd _ZN8CryptoPP16IteratedHashBaseIjNS_25MessageAuthenticationCodeEE18HashMultipleBlocksEPKjm + 237453
[13] 14 residue 0x000000010ca31612 residue + 280082
[14] 15 residue 0x000000010cb53192 _ZN8CryptoPP16IteratedHashBaseIjNS_25MessageAuthenticationCodeEE18HashMultipleBlocksEPKjm + 210002
[15] 16 residue 0x000000010ca92e18 _ZN4asio6detail28reactive_socket_send_op_baseINS_15const_buffers_1EE10do_performEPNS0_10reactor_opE + 150952
[16] 17 residue 0x000000010cb06814 _ZNK8CryptoPP4OAEPINS_4SHA1ENS_10P1363_MGF1EE6NewMGFEv + 135508
[17] 18 residue 0x000000010ca918e3 _ZN4asio6detail28reactive_socket_send_op_baseINS_15const_buffers_1EE10do_performEPNS0_10reactor_opE + 145523
[18] 19 residue 0x000000010ca91a3b _ZN4asio6detail28reactive_socket_send_op_baseINS_15const_buffers_1EE10do_performEPNS0_10reactor_opE + 145867
[19] 20 residue 0x000000010ca2fe8a residue + 274058
[20] 21 residue 0x000000010ca28c07 residue + 244743
[21] 22 residue 0x000000010ca6643b residue + 496699
[22] 23 residue 0x000000010ca50c63 residue + 408675
[23] 24 residue 0x000000010ca4dd0d residue + 396557
[24] 25 residue 0x000000010ca4d2c9 residue + 393929
[25] 26 residue 0x000000010c9f0953 residue + 14675
[26] 27 libdyld.dylib 0x00007fff73a99115 start + 1
[27] 28 ??? 0x0000000000000003 0x0 + 3
If we have long running log queue and following things happen all at once:
all the subsequent request in next backlog process will fail
This is very rare situation and we only found this out after crazy load testing and really hammering the server.
Currently demo docs are not valid as we have shut down demo server.
Change the docs so user can create their own fully function demo server (both closed and open server)
also clean /demo
directory to properly demonstrate the usage of the server.
Log rotator schedules from when server is started. i.e, if server started at 9:36am the next daily rotation will rotate at 10:00 on that day (it should be next day instead) and subsequent rotations will also be 10:00 each day
This is not expected behaviour as user may expect daily logs to be only archived the next day and not on the next lowest denominator run (i.e, HOURLY)
The hourly rotation rounds off to next 00 minutes instead of the start of server which is a plus and makes this fix easier ✅
In above scenario the daily rotation should be at next nearest 00:00 and then continue on wards on this time
Same thing should happen for weekly, monthly and yearly tasks
v1.2.3 (all previous versions)
22:30:51,352 [LogDispatcher] [FATAL] CRASH HANDLED; Application has crashed due to [SIGSEGV] signal
======= Backtrace: =========
[1] 2 residue 0x000000010c5c690b _Z23generalTerminateHandlerib + 267
[2] 3 libsystem_platform.dylib 0x00007fff6865ef5a _sigtramp + 26
[3] 4 ??? 0x0000000000000002 0x0 + 2
[4] 5 residue 0x000000010c66d08d _ZNSt3__16__treeINS_12__value_typeINS_12basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEEEN8nlohmann10basic_jsonINS_3mapENS_6vectorES7_bxydS5_NS8_14adl_serializerEEEEENS_19__map_value_compareIS7_SE_NS_4lessIS7_EELb1EEENS5_ISE_EEE7destroyEPNS_11__tree_nodeISE_PvEE + 29
[5] 6 residue 0x000000010c66d08d _ZNSt3__16__treeINS_12__value_typeINS_12basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEEEN8nlohmann10basic_jsonINS_3mapENS_6vectorES7_bxydS5_NS8_14adl_serializerEEEEENS_19__map_value_compareIS7_SE_NS_4lessIS7_EELb1EEENS5_ISE_EEE7destroyEPNS_11__tree_nodeISE_PvEE + 29
[6] 7 residue 0x000000010c66d08d _ZNSt3__16__treeINS_12__value_typeINS_12basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEEEN8nlohmann10basic_jsonINS_3mapENS_6vectorES7_bxydS5_NS8_14adl_serializerEEEEENS_19__map_value_compareIS7_SE_NS_4lessIS7_EELb1EEENS5_ISE_EEE7destroyEPNS_11__tree_nodeISE_PvEE + 29
[7] 8 residue 0x000000010c66d099 _ZNSt3__16__treeINS_12__value_typeINS_12basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEEEN8nlohmann10basic_jsonINS_3mapENS_6vectorES7_bxydS5_NS8_14adl_serializerEEEEENS_19__map_value_compareIS7_SE_NS_4lessIS7_EELb1EEENS5_ISE_EEE7destroyEPNS_11__tree_nodeISE_PvEE + 41
[8] 9 residue 0x000000010c66d099 _ZNSt3__16__treeINS_12__value_typeINS_12basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEEEN8nlohmann10basic_jsonINS_3mapENS_6vectorES7_bxydS5_NS8_14adl_serializerEEEEENS_19__map_value_compareIS7_SE_NS_4lessIS7_EELb1EEENS5_ISE_EEE7destroyEPNS_11__tree_nodeISE_PvEE + 41
[9] 10 residue 0x000000010c66d08d _ZNSt3__16__treeINS_12__value_typeINS_12basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEEEN8nlohmann10basic_jsonINS_3mapENS_6vectorES7_bxydS5_NS8_14adl_serializerEEEEENS_19__map_value_compareIS7_SE_NS_4lessIS7_EELb1EEENS5_ISE_EEE7destroyEPNS_11__tree_nodeISE_PvEE + 29
[10] 11 residue 0x000000010c66cf78 _ZN8nlohmann10basic_jsonINSt3__13mapENS1_6vectorENS1_12basic_stringIcNS1_11char_traitsIcEENS1_9allocatorIcEEEEbxydS7_NS_14adl_serializerEED2Ev + 56
[11] 12 residue 0x000000010c66cfad _ZN8nlohmann10basic_jsonINSt3__13mapENS1_6vectorENS1_12basic_stringIcNS1_11char_traitsIcEENS1_9allocatorIcEEEEbxydS7_NS_14adl_serializerEED2Ev + 109
[12] 13 residue 0x000000010c61d833 _ZN7residue7RequestD2Ev + 83
[13] 14 residue 0x000000010c619581 _ZN7residue17LogRequestHandler19processRequestQueueEv + 4449
[14] 15 residue 0x000000010c61d8e8 _ZNSt3__114__thread_proxyINS_5tupleIJNS_10unique_ptrINS_15__thread_structENS_14default_deleteIS3_EEEEZN7residue17LogRequestHandler5startEvE3$_0EEEEEPvSB_ + 152
[15] 16 libsystem_pthread.dylib 0x00007fff686686c1 _pthread_body + 340
[16] 17 libsystem_pthread.dylib 0x00007fff6866856d _pthread_body + 0
[17] 18 libsystem_pthread.dylib 0x00007fff68667c5d thread_start + 13
Provide CLI and admin request to be able to reload the registered logger configuration
19:03:02,606 [LogHandler] [vTRACE] [session.cc:86] Adding bytes
19:03:02,608 [LogDispatcher] [vDEBUG] [log-request-handler.cc:88] Pausing schedule for client integrity
19:03:02,609 [LogHandler] [vTRACE] [session.cc:75] Received: 522 bytes
19:03:02,609 [LogHandler] [vTRACE] [session.cc:102] Read bytes: [..snip] [size: 522]
19:03:02,609 [LogHandler] [vDEBUG] [session.cc:171] Sending {r:0}
19:03:02,609 [LogHandler] [vTRACE] [session.cc:86] Adding bytes
19:03:02,609 [LogDispatcher] [vCRAZY] [log-request-handler.cc:100] -----============= [ BEGIN ] =============-----
19:03:02,610 [LogDispatcher] [vDEBUG] [request-handler.h:144] Raw request: [..snip]
19:03:02,610 [LogDispatcher] [vDEBUG] [request-handler.cc:58] Client: [..snip]
19:03:02,610 [LogDispatcher] [vCRAZY] [request-handler.cc:59] IV: [..snip]
19:03:02,610 [LogDispatcher] [vCRAZY] [request-handler.cc:66] Data (base64): [..snip]
19:03:02,610 [LogDispatcher] [vCRAZY] [request-handler.cc:69] Ripe command: [..snip]
19:03:02,610 [LogDispatcher] [vCRAZY] [request-handler.cc:76] Plain request: [..snip]
19:03:02,610 [LogDispatcher] [vTRACE] [request-handler.h:148] Decryption finished (b64): [..snip]
19:03:02,610 [LogDispatcher] [vTRACE] [request-handler.h:159] Decompressing: [..snip]
19:03:02,611 [LogDispatcher] [vTRACE] [request-handler.h:170] Decompression finished (raw): [{... (removed)}]
19:03:02,612 [LogDispatcher] [vDEBUG] [log-request-handler.cc:115] Is bulk? true
19:03:02,612 [LogDispatcher] [vDEBUG] [log-request-handler.cc:124] Request client: 0x10065cfa8
19:03:02,612 [LogHandler] [vTRACE] [session.cc:75] Received: 522 bytes
19:03:02,612 [LogDispatcher] [vDEBUG] [log-request-handler.cc:206] Force check: true, clientRef: 0x70000f9729a0, *clientRef: muflihun00102030, bypassChecks: false
19:03:02,612 [LogHandler] [vTRACE] [session.cc:102] Read bytes: 29277885c8e3830c20fb91fab830194b:muflihun00102030:zaAqRKANlz9SElDUXg6ePbeykddpAyJjxC3TEL/yutU5HIryqLpN1xmsEncDHizpniIeaCIY7kfUjuzTaXwYJC0dxHxlssq5Lm7PzgPCyyxa7POh0quW1zooTSlBMvUiuCWamAdfzz3WPPjjbMSyvqtgMmbFIhs4qMMMt3kKZbli8kk9Hmv5XV8Ltz+/tLX4ppiOn6qSFh/pIQ0LNaBVK0wHXLfGO+AN58vc1D50JT8vA00XUjkzNNmeasekIntZDoRut/WhgqVijoUsT27sY9WzrsdaLuIJIAUhtewqvAGgyeplI+lucgHWtWKVR+HQQ887giQwW6+80FrU0yIZ/yEhDnFbCul3Zc9L3vuuBHvrAgjYvDPkA4MZPwBkFWSh6anUbJ5zsNgEnMP2tdDI+ZLnSVfvQ+ZNEPeVXYE15la/XVPoe0/b/2MqY3HPUrYEtTJONqT/XrZKl1HXbqxp4w== [size: 522]
19:03:02,613 [LogDispatcher] [vDEBUG] [client.cc:65] Checking token 37953058 (client [muflihun00102030])
Because configuration::save uses original json
We will remove token server as we do not need it after we removed plain log request support
This decision is subject to change. We need to further think on whether we really want to do it or it's necessary.
[This will break existing client libraries as we will not have token port available]
This has following impacts
requires_timestamp
as it will always be requiredLogRequest::validateTimestamp
to check for time stampstoken_age
, max_token_age
, access codes concepts, requires_token
can all be removed from code baseThis is subject to issue 613 on Easylogging++. That needs to be fixed before this should be worked on otherwise we may run in to race condition
This is an improvement to prevent one client blocking other client's logs in case of “rush hour”
We will have separate queue for unknown clients and one queue for each known client initialized at the startup
This is a security concern + extra instructions for potentially unused feature
We will remove it in 1.5.0
JSON parser unable to parse string with double quotes. This must be fixed as messages may contain double quotes
v1.4.3
Since 1.6.0 (the custom queue processor feature) we didn't add custom processors for dynamically added clients
If log processing is running behind and one of the clients are past their age and client integrity task runs in the mean time, it will remove the client from registry. This will cause the log to fail with invalid message [log-request-handler.cc:107] Failed: Client not connected yet
Currently the situation is handled and client is temporarily brought alive if client is DEAD but still registered.
Log processing should be successful for such dead client because it passed the original validation (at time of connection and token retrieval)
v1.x.x
Request will crash the server
printf "{\"type\":[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[,\"rsa_public_key\":\"`cat client-256-public.pem | ripe -e --base64`\"}\r\n\r\n" | nc `cat host` 8777
v1.4.2
JSON for Modern C++ is very good library and we love it. but it has features that we do not need and we needed very basic JSON implementation. We improved compiled time with this
We have tested new library and it works beautifully to suit our needs. We will remove old library's code from our codebase
We need to ensure reloading config reconfigures existing loggers
We need to re-think the way when we respond back to client for the log request. We hold the shared pointer to the session which causes client to potentially wait until the log is processed (unless _close
flag is on)
When log rotation truncates the file and file is being created. Last log message seen is
22:51:56,265 [AdminHandler] [INFO] FSR: Prepending 'level' format specifier in filename for logger [sample-app] as we have multiple filenames for levels
22:51:56,265 [AdminHandler] [INFO] FSR: [/tmp/logs/sample-app.log] => [/tmp/logs/backups/sample-app/] as [global-2018.log]
22:51:56,265 [AdminHandler] [INFO] FSR: Result: [/tmp/logs/backups/sample-app/] [2018.tar.gz] with [3] items
22:51:56,265 [AdminHandler] [vDETAILS] [log-rotator.cc:272] Ignoring rotating empty file /tmp/logs/sample-app-verbose.log
22:51:56,265 [AdminHandler] [vDETAILS] [log-rotator.cc:272] Ignoring rotating empty file /tmp/logs/sample-app-debug.log
22:51:56,265 [AdminHandler] [vDETAILS] [log-rotator.cc:261] Rotating [/tmp/logs/sample-app.log] => [/tmp/logs/backups/sample-app/global-2018.log] (1.2GB)
22:51:56,265 [LogDispatcher] [vDEBUG] [log-request-handler.cc:185] Force check: 0, clientRef: 0x7000098bddf0, *clientRef: muflihun00102030, bypassChecks: 1
22:51:56,265 [LogDispatcher] [ERROR] File not found [/tmp/logs/sample-app.log] [Logger: sample-app]. Creating...
22:51:56,265 [LogDispatcher] [INFO] Accessing file...
The next line is updating permission but there is no lock required from this point onwards so not sure why is this happening
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.