crashctl's Introduction

crashctl

A simple tool for crashing server diagnosis. A replacement for journalctl --list-boots with additional and more readable information.

$ crashctl
Distribution        : Debian GNU/Linux 10 (buster)
Kernel              : 4.19.0-23-amd64 #1 SMP Debian 4.19.269-1 (2022-12-20)
Current boot        : 555ebf2e-5cd0-4f52-b6fe-a4734248cc82
Scaled load         : 0.24 0.25 0.23
System installed    : Wed Feb  1 09:41:10 UTC 2023
System started      : Mon Feb 13 11:33:08 UTC 2023
Uptime              : up 4:52
Running processes   : 401
kdump               : kdump-tools not installed. Try: apt install kdump-tools
Boot First message        Last message             Uptime       Reboot/Crash
-------------------------------------------------------------------------------------
-1   2023-02-07 06:29:00 UTC 2023-02-13 11:31:08 UTC  6d 05:02:08  reboot (SIGTERM)
0    2023-02-13 11:33:16 UTC 2023-02-13 16:25:43 UTC  0d 04:52:27  running

Install

The script is written Bash, it might work on any distribution using Systemd.

wget https://raw.githubusercontent.com/deric/crashctl/main/crashctl.sh -O /usr/local/bin/crashctl

Asumes existence of basic Unix utils as cat, awk, stat

Following utils might require root priviledges:

journalctl
tune2fs

Usage

See crashctl -h for usage.

With kdump-tools installed each session might be matched to a crash log:

-2   2023-09-08 11:35:19 UTC   2023-09-08 11:35:32 UTC  0d 00:00:13  kernel panic. See /var/crash/202309081135
-1   2023-09-08 11:38:37 UTC   2023-09-08 12:45:32 UTC  0d 01:06:55  CRASH?
0    2023-09-08 13:39:29 UTC   2023-09-08 21:31:53 UTC  0d 07:52:24  running

Advanced usage

Detecting crash is not always reliable sometimes kernel might crash without logging a message or the power could be cut, etc. Reboot or shutdown sequence can be found in logs.

$ crashctl --boots --utc
Boot First message             Last message             Uptime       Reboot/Crash
-------------------------------------------------------------------------------------
-11  2022-12-05 20:43:53 UTC   2022-12-05 20:52:00 UTC  0d 00:08:07  reboot (SIGTERM)
-10  2022-12-06 07:56:01 UTC   2022-12-06 15:14:36 UTC  0d 07:18:35  CRASH?
-9   2022-12-07 12:28:07 UTC   2022-12-10 16:33:43 UTC  3d 04:05:36  reboot (SIGTERM)
-8   2022-12-12 08:56:05 UTC   2022-12-18 08:18:40 UTC  5d 23:22:35  CRASH?
-7   2022-12-18 08:32:27 UTC   2022-12-25 10:54:03 UTC  7d 02:21:36  reboot (SIGTERM)
-6   2022-12-28 10:51:54 UTC   2022-12-29 12:12:32 UTC  1d 01:20:38  Power key pressed, but ignored
-5   2023-01-02 08:45:54 UTC   2023-01-06 08:05:01 UTC  3d 23:19:07  CRASH?
-4   2023-01-06 10:07:00 UTC   2023-01-12 10:01:25 UTC  5d 23:54:25  Power key pressed, but ignored
-3   2023-01-12 10:04:36 UTC   2023-01-28 14:07:19 UTC  16d 04:02:43 reboot (SIGTERM)
-2   2023-01-30 08:43:42 UTC   2023-01-31 07:27:26 UTC  0d 22:43:44  reboot (SIGTERM)
-1   2023-02-02 12:41:51 UTC   2023-02-04 13:16:19 UTC  2d 00:34:28  reboot (SIGTERM)
0    2023-02-06 03:12:01 UTC   2023-02-13 18:17:52 UTC  7d 15:05:51  running

crashctl's People

Contributors

Stargazers

Watchers

crashctl's Issues

Script not working properly due to missing commans and and invalid arguments?

Hello I'm trying to diagnose my server reboots (running on Proxmox 8.x lastest)
(Formatting is broken by GitHub, sorry)

The output is broken:

  root@sm836:~# ./crashctl.sh
 ./crashctl.sh: line 40: lsb_release: command not found
 Distribution        :
 Kernel              : 6.2.16-4-pve #1 SMP PREEMPT_DYNAMIC PVE 6.2.16-5 (2023-07-14T17:53Z)
 Current boot        : e507d538-affc-49e6-a209-25ea8b78be5b
 Scaled load         : 0.09 0.11 0.07
 System installed    : Sun Jul  2 00:00:00 -05 2023
 System started      : Mon Jul 24 17:02:06 -05 2023
 Uptime              : up 11 min
 Running processes   : 621
 kdump               : kdump-tools not installed. Try: apt install kdump-tools
 Boot First message             Last message             Uptime       Reboot/Crash
 -------------------------------------------------------------------------------------
 Failed to add match 'IDX': Invalid argument
 Failed to add match 'IDX': Invalid argument
 Failed to add match 'IDX': Invalid argument
 ./crashctl.sh: line 190: ary[7]: unbound variable

Journalctl output:

 root@sm836:~# journalctl --list-boots
 IDX BOOT ID                          FIRST ENTRY                 LAST ENTRY
 -12 d694f3a6a4e04581853059104036c961 Sun 2023-07-02 21:47:46 -05 Sun 2023-07-02 22:40:42 -05
 -11 7d3971f8b9db4bb7ab83522027155d6d Sun 2023-07-02 22:46:59 -05 Sun 2023-07-02 22:50:33 -05
 -10 f6859b270aff4d1d887eab4216120c0f Sun 2023-07-02 22:56:49 -05 Sun 2023-07-02 23:42:44 -05
  -9 88fe8a6844bb4f95bdbe38349cd167a9 Sun 2023-07-02 23:48:48 -05 Mon 2023-07-03 01:10:14 -05
  -8 c3e8a5c5476b4c90b8a0e7cb9e4610fe Mon 2023-07-03 01:16:35 -05 Fri 2023-07-14 12:51:16 -05
  -7 ebbd8447ddd4452c8a6080e3025572b9 Fri 2023-07-14 14:10:51 -05 Tue 2023-07-18 07:03:01 -05
  -6 0af22536ffe9477999697c811be1b094 Tue 2023-07-18 07:09:52 -05 Tue 2023-07-18 14:41:46 -05
  -5 d351e8f4df8049c48d4e524fc68d725e Tue 2023-07-18 14:47:46 -05 Tue 2023-07-18 16:45:11 -05
  -4 77352396a99344eca328004dcf19bf51 Tue 2023-07-18 16:51:46 -05 Wed 2023-07-19 11:13:30 -05
  -3 fc52604b477a4f67a2941e46ed756156 Wed 2023-07-19 11:20:24 -05 Sun 2023-07-23 03:37:08 -05
  -2 a7a009d4c7934755815ee3f7148c9ad6 Sun 2023-07-23 03:43:55 -05 Mon 2023-07-24 12:53:29 -05
  -1 774606b4399c409788470d91d78a89c6 Mon 2023-07-24 13:00:27 -05 Mon 2023-07-24 16:56:47 -05
   0 e507d538affc49e6a20925ea8b78be5b Mon 2023-07-24 17:03:13 -05 Mon 2023-07-24 17:17:36 -05

An alternative to lsb-release (not every distro has it by default):

root@sm836:~# cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
NAME="Debian GNU/Linux"
VERSION_ID="12"
VERSION="12 (bookworm)"
VERSION_CODENAME=bookworm
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

Recommend Projects