Couchdb stoped writing to the journald logs

[tahirshak]

Couchdb stoped writing to the journald logs

[nickva]

Thank you for your report @tahirshak.

Is there any log emitted right before that?

After that happens does couchdb still work? Can you use the Fauxton interface or make API calls to it?

Does its resources usage seem to change during that time (uses more CPU/memory or less?).

[tahirshak]

Hi @nickva , Please see the response to your questions.

Q: Is there any log emitted right before that?

Looking at the logs leading to when couchdb stopped writing to the journal, in just 5 minutes it emitted about 45k logs entries. Most of those messages are about the indexer such as Starting index update for db: shards and Index update finished for db: shards .

Q: After that happens does couchdb still work? Can you use the Fauxton interface or make API calls to it?

Couchdb still works and functioning, we can login to Fauxton and run the curl commands.

Q: Does its resources usage seem to change during that time (uses more CPU/memory or less?).

Yes, the CPU spiked little bit to 40% and the memory spiked as well to 15%.

[nickva]

@tahirshak thank you for responding. That's interesting about the CPU spiking a bit afterwards. I still don't quite have a clue what might be going on.

In production I only have experience with the syslog writer. That one never seems to behave this way. The journald log writer just seems to write to standard_error c2ff7da but I can't think why it would stop. Wonder if a log gets full or can't be rotated then standard_error would lock up?

Would it be possible for you try a syslog writer and have a system rsyslog package installed to see if it would behave the same way?

[nickva]

Perhaps it's similar to coreos/bugs#990, a systemd issue?

If it's triggered by log volume can try lowering the log level to error: https://docs.couchdb.org/en/stable/config/logging.html#log/level

I don't know if you want to dive in and debug journald/systemd but I owuld to bypass systemd altogether and log to a file or use rsyslog and use the syslog backend: https://docs.couchdb.org/en/stable/config/logging.html#log/syslog_host