Comments (10)
More info: Version is 3.0.3
# ./check_raid.pl -p mdstat -V
check_raid Version 3.0.3
Running Debian v7.2
# cat /etc/debian_version
7.2
from nagios-plugin-check_raid.
you probably want --resync=WARNING
option
from nagios-plugin-check_raid.
I'll do that, but I'm really in a worse state than that :). I'm actually in a degraded mode, and if there is some way to reflect that status, that would be great. If I perform a mdadm -detail, I get a more accurate state:
# mdadm --detail /dev/md0
/dev/md0:
Version : 0.90
Creation Time : Mon Sep 28 23:50:50 2009
Raid Level : raid6
Array Size : 5860543744 (5589.05 GiB 6001.20 GB)
Used Dev Size : 1465135936 (1397.26 GiB 1500.30 GB)
Raid Devices : 6
Total Devices : 6
Preferred Minor : 0
Persistence : Superblock is persistent
Update Time : Tue Nov 19 10:44:38 2013
State : clean, degraded, recovering
Active Devices : 4
Working Devices : 6
Failed Devices : 0
Spare Devices : 2
Layout : left-symmetric
Chunk Size : 4K
Rebuild Status : 23% complete
UUID : c0a45de6:f2b14d16:46611cb4:29fb6250 (local to host <hostname>)
Events : 0.457232
Number Major Minor RaidDevice State
0 8 17 0 active sync /dev/sdb1
1 8 32 1 active sync /dev/sdc
2 8 49 2 active sync /dev/sdd1
3 8 97 3 active sync /dev/sdg1
6 8 65 4 spare rebuilding /dev/sde1
7 8 81 5 spare rebuilding /dev/sdf1
I'm running RAID 6 so I can lose two drives. I'm not sure what the status of the array will be if I lose 1 drive, and therefore how the plugin will behave. I'll do some more tests this week after the recovery is complete, as I need to take one of my drives and partition it appropriately.
BTW, thanks for providing this plugin, it seems VERY comprehensive.
from nagios-plugin-check_raid.
does mdadm use some other data than /proc/mdstat?
theoretically it would be possible to know if it's degraded, i.e you know it's raid6 and two disks out (UUUU__), then it's degraded. but i don't want to take responsibility to calculate that wrong :(
currently there's no mdadm driver, perhaps i should create it
ps: thanks, i recently updated mdstat plugin to parse /proc/mdstat exactly as kernel writes it. well, not deep to each raid variation (each raidX can write own output), but at least until the external device part.
from nagios-plugin-check_raid.
current code does:
if (my($action, $perc, $eta, $speed) =
m{(resync|recovery|check|reshape)\s+=\s+([\d.]+%) \(\d+/\d+\) finish=([\d.]+min) speed=(\d+K/sec)}) {
can i safely say that resync
and check
set state with --resync=
option, but recovery
and reshape
always trigger critical state?
from nagios-plugin-check_raid.
That's a good question, as this is the first major re-work I have done on this array in about 2 years. I want to see how the array behaves under certain conditions before I'd make a recommendation. Here's my story:
I had 5x1.5 TB RAID 5 array. A little over 1 week ago, I lost a drive. The Nagios plugin I was using didn't report the failure, I was lucky to discover the state. I ordered two new drives. I replaced one and started the rebuild, which went successfully. Once the rebuild was complete, I decided to convert (reshape) to a RAID 6, using the other drive as the extra parity drive. I also replaced the Nagios plugin I had with yours. During the reshape, about 24 hours apart, I lost two drives (not sure what happened, power issue or controller issue), and that caused the array to fail. I spent a few hours getting the array back up, and the reshape continued, but Nagios never reported anything other than OK when the array was up.
Suffice it to say, this weekend was a crash course in Linux Software RAID management. mdadm is the management utility for Linux Software RAID, so I'm not sure how it gathers its data, but it's entirely possible for the RAID information on each of the drives to be out of sync in a failure state. /proc/mdstat does seem to be an accurate summary, however.
When I first started the reshape, I was redundant, as I had 4+1 and another parity drive being added. I wouldn't consider the reshape to be critical in this state. Maybe warning, as the array is being modified administratively. When I lost my first drive and was in a degraded/non-redundant state, I'd consider that to be critical, regardless of the activity.
The other reason why I'm hesitant to state that recovery would always be critical is because I am running RAID 6 (4+2). If I lose a drive, I'll be 4+1. I'll still have parity/redundancy, so it wouldn't be critical (to me at least). I'd consider the loss of 1 drive in RAID 6 to be a warning condition.
I'm wondering what state the array will report in that slightly degraded condition. I can send you output of both /proc/mdstat and mdadm --detail when I perform that test later this week.
Sorry for the rambling comment, thanks for your attention and support!
from nagios-plugin-check_raid.
The rebuild completed successfully, and I am doubly redundant. I removed one of the disks to see how the RAID software would respond:
# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid6 sdb1[0] sdf1[5] sde1[4] sdg1[3] sdd1[2]
5860543744 blocks level 6, 4k chunk, algorithm 2 [6/5] [U_UUUU]
/proc/mdstat didn't seem to show the degraded state. Then I checked with mdadm:
# mdadm --detail /dev/md0
/dev/md0:
Version : 0.90
Creation Time : Mon Sep 28 23:50:50 2009
Raid Level : raid6
Array Size : 5860543744 (5589.05 GiB 6001.20 GB)
Used Dev Size : 1465135936 (1397.26 GiB 1500.30 GB)
Raid Devices : 6
Total Devices : 5
Preferred Minor : 0
Persistence : Superblock is persistent
Update Time : Wed Nov 20 20:52:56 2013
State : clean, degraded
Active Devices : 5
Working Devices : 5
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 4K
UUID : c0a45de6:f2b14d16:46611cb4:29fb6250 (local to host <hostname>)
Events : 0.461176
Number Major Minor RaidDevice State
0 8 17 0 active sync /dev/sdb1
1 0 0 1 removed
2 8 49 2 active sync /dev/sdd1
3 8 97 3 active sync /dev/sdg1
4 8 65 4 active sync /dev/sde1
5 8 81 5 active sync /dev/sdf1
This seemed to be more accurate. I then partitioned the drive I just removed, added it back to the array, and checked the RAID status:
# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid6 sdc1[6] sdb1[0] sdf1[5] sde1[4] sdg1[3] sdd1[2]
5860543744 blocks level 6, 4k chunk, algorithm 2 [6/5] [U_UUUU]
[>....................] recovery = 0.2% (3563496/1465135936) finish=1638.2min speed=14868K/sec
unused devices: <none>
I then checked with mdadm, which again seemed to return accurate results:
# mdadm --detail /dev/md0
/dev/md0:
Version : 0.90
Creation Time : Mon Sep 28 23:50:50 2009
Raid Level : raid6
Array Size : 5860543744 (5589.05 GiB 6001.20 GB)
Used Dev Size : 1465135936 (1397.26 GiB 1500.30 GB)
Raid Devices : 6
Total Devices : 6
Preferred Minor : 0
Persistence : Superblock is persistent
Update Time : Wed Nov 20 20:57:03 2013
State : clean, degraded, recovering
Active Devices : 5
Working Devices : 6
Failed Devices : 0
Spare Devices : 1
Layout : left-symmetric
Chunk Size : 4K
Rebuild Status : 0% complete
UUID : c0a45de6:f2b14d16:46611cb4:29fb6250 (local to host <hostname>)
Events : 0.461194
Number Major Minor RaidDevice State
0 8 17 0 active sync /dev/sdb1
6 8 33 1 spare rebuilding /dev/sdc1
2 8 49 2 active sync /dev/sdd1
3 8 97 3 active sync /dev/sdg1
4 8 65 4 active sync /dev/sde1
5 8 81 5 active sync /dev/sdf1
From what I can tell, with a RAID 6 configuration, /proc/mdstat won't show a degraded state until a spare is rebuilding, but mdadm --detail will provide an accurate status of the array.
Thanks again for looking into this!
from nagios-plugin-check_raid.
i'm not sure what to do here, so i just added test data for mdstat from last posts
from nagios-plugin-check_raid.
should add here probably new "driver", processing with mdadm
binary
from nagios-plugin-check_raid.
closing for now
from nagios-plugin-check_raid.
Related Issues (20)
- arcconf: bbu monitoring for flash+battery/supercap adaptec modules HOT 1
- arcconf: doesn't handle logical device status: Impacted (Build/Verify with fix : 90 %) HOT 2
- Allow to ignore plugins
- No remote servers RAID status: check_raid UNKNOWN HOT 16
- Possible to hide "OK" results or just showing "OK" and not all devices? HOT 4
- Error runing plug-in on debian 10 HOT 10
- remote monitor HOT 1
- Use of StorCli instead of MegaCli HOT 5
- [bug] with parsing mdadm output => Unexpected parse
- Sudo error on centos
- WARNING: megacli:[Volumes(1): DISK0.0:Optimal, WriteCache:DISABLED HOT 2
- arcconf: parse error HOT 6
- Unexpected parse at ./check_raid.pl line 3578, <$fh> line 6. HOT 1
- HP ssacli - consider predictive failure information HOT 6
- HP ssacli - rebuilding shouldn't perhaps still be warning state? HOT 6
- HP ssacli -
- Error with P440AR RAID controller. How to solve, what to change? Please advice. HOT 6
- "NOT PARSED" and "Can't locate object method" errors HOT 2
- Feature Request mdstat: Warn when auto-read-only array detected
- "sudo: a password is required" on Ubuntu 20.0.4 HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from nagios-plugin-check_raid.