Comments (12)
Hi @tt4g , we are not passing the sink to another program (just another thread in a dylib), and I doubt the write to pipe can cause a hang up to minutes. In addition, we use PTP for clock sync on the machine so that also rules out it being caused by massive drift in timestamp; to clarify, we noticed the hang through other out-of-band channels and not just based on the timestamp.
I have not been able to attach to the program at the exact time when the hang happens yet, since this is fairly hard to repro and the hang often resolves itself in a few minutes. I am going to try to reproduce this, and perhaps upgrading the spdlog version may resolve the issue as well?
Thank you :)
from spdlog.
I am not familiar with the GNU screen implementation, but I suspect that the output buffer is full and write(2)
is blocked.
I found a very similar question: https://unix.stackexchange.com/questions/325588/prevent-gnu-screen-from-blocking-standard-out
from spdlog.
If spdlog is called many times in a short period of time, it is expected to consume time in critical sections used for thread-safety.
However, that is not necessarily the reason for a gap of several minutes in the log time.
Is there any reason to be certain that it is being caused by spdlog?
Is there some basis for this, such as “the problem did not reproduce itself when using a single sink.”
from spdlog.
Assuming that the hang is in spdlog, it is likely that spdlog hangs because it is taking time to acquire a lock on the std::mutex
that it uses for thread-safety. Another possibility is that the operation to write the log to disk is taking a long time.
However, these are OS processes, so spdlog would not work around the problem.
These investigations are not easy, so first we need to be sure that it is indeed spdlog that is hanging.
from spdlog.
@tt4g thanks for the response!
Is there any reason to be certain that it is being caused by spdlog?
I believe this is the only explanation since the hang happens between back-to-back spdlog invocation with nothing in between.
Do you see anything obviously wrong in the way spdlog is initialized and used, in the snippet above? I assume spd::logger
is thread safe in this setup?
I wasn't able to find documentation about it, the only remark I found on wiki is about basic_logger_mt
, which does support multi-threading, but only writes output to file.
from spdlog.
Do you see anything obviously wrong in the way spdlog is initialized and used, in the snippet above? I assume
spd::logger
is thread safe in this setup?
It is not possible to determine from the snippet whether static_cast<spdlog::level::level_enum>(PLN_LOG_LEVEL)
is a safe operation.
Otherwise, it seems to be calling the thread-safe API correctly.
If the log times are off on two logging API calls, you need to find out what is hanging.
I am using two sinks, can I reproduce the problem using only one of the sinks?
If the problem only occurs with one of the sinks, then there may be a problem with slow writes to that sink's output destination, etc.
NOTE that spdlog gets the current time from the system clock, so if the system clock changes, the timestamp printed in the log may also shift significantly.
from spdlog.
FYI.
If you are passing STDOUT (stdout_color_sink_mt) to another program, it is possible that the hang is caused by writing to stdout.
For example, if you use command line pipe, UNIK Like OS provides a buffer for the pipe, but if the pipe buffer is consumed slowly, the buffer will be full and write(2)
will block until the buffer is consumed.
https://man7.org/linux/man-pages/man7/pipe.7.html
A pipe has a limited capacity. If the pipe is full, then a write(2) will block or fail, depending on whether the O_NONBLOCK flag is set (see below).
from spdlog.
I hope the upgrade does not reproduce the problem, as it is extremely difficult to determine the cause of a problem that does not reproduce.
from spdlog.
Hi @tt4g , as you suggested above I looked more into the possibility of a blocking pipe write.
During some runs we are putting the application in a screen session, and as it turns out that if the screen session is put in scroll mode (Ctrl+a q
) or (Ctrl+a [
) it will block the output pipe that the console sink is writing to, after the pipe buffer is full. This seems to effectively block spdlog for indefinitely period of time. I was able to reproduce this in a toy example.
I'm guessing we can avoid this problem by using the async logger, which will probably still block the background thread in this case but at least won't hang the main application thread.
Do you think this is in line with your expectation/suggestion?
from spdlog.
Thanks! I assume this blocking behavior is going to happen with async logger as well (blocking the bg thread)?
from spdlog.
This problem is reproduced using the aync logger.
This is because the Linux Kernel blocks writes unless the STDOUT buffer is consumed.
from spdlog.
@tt4g Yes that's what I would expect. Since this is not an issue with spdlog, closing this ticket. Thank you very much for the quick turnaround and all the help!! All the best.
from spdlog.
Related Issues (20)
- spdlog 1.14.1 fails to build with fmt 11.0.1 HOT 2
- Log rotation does not work if the log file has multiple owner processes
- i can't build it about Spalog, need help~ HOT 1
- compiler error on vs2015 HOT 5
- why the symbol of stopwatch not in libspdlog.dylib HOT 1
- ANSI color sink on Windows
- Resource deadlock would ocur HOT 6
- Undefined symbol error for spdlog::details::os::utf8_to_strbuf on Windows HOT 1
- build with SPDLOG_FMT_EXTERNAL_HO error HOT 2
- Windows linker warnings disabled HOT 1
- How do I log a std::iostream? HOT 2
- [MSVC 17.10.5] spdlog::sinks::base_sink<Mutex>::set_pattern() crash HOT 5
- For non-SPDLOG_HEADER_ONLY builds enable export attributes for symbols HOT 9
- Enums do not work. HOT 1
- Why customized logger can't print function name? HOT 3
- When initializing, it prompts "simple log instance" HOT 1
- spdlog does not print in python async function HOT 1
- error: call to non-‘constexpr’ function ‘long int sysconf(int)’ HOT 1
- I found that we can use python format style: HOT 2
- Setting compile time log level seems broken. HOT 10
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from spdlog.