spearfoot/freenas-scripts

Handy shell scripts for use on FreeNAS servers

freenas-scripts's Issues

Unable to decrypt the config backup

I used the save_config_enc.sh script to create an encrypted back. Then I used the example openssl command to try to decrypt it. However, I just get an error:

root@freenas ~/FreeNAS-scripts # ./save_config_enc.sh 
Backup configuration database file: /mnt/storage/backup/freenas/freenas-FreeNAS-11.3-RELEASE-c0e049a7fa-20200207121125.db
Configuration file copied with status 0

root@freenas ~ # openssl enc -d -aes256 -pass file:config_passphrase -in /mnt/storage/backup/freenas/freenas-FreeNAS-11.3-RELEASE-c0e049a7fa-20200207121125.db -out decrypted.db
bad magic number

3ware controller

so for anyone who happens to stumble on this it wont work with a 3ware controller
solution
everyline that has smartctl .... /dev/$drive add twa0 -d 3ware, after dev
the line will read .../dev/twa0 -d 3ware,${drive}
then uncomment all of the 3 methods for determining drives and replace with drives="0 1 2 3 4 5 ....."

Output of scripts show some errors

Hey thanks for these scripts, i hope we can fix them for the most current release.

root@freenas:~ # bash get_hdd_temp.sh
=== CPU (8) ===
sysctl: unknown oid 'dev.cpu.0.temperature'
/tmp/1: line 72: [: : integer expression expected
CPU  0:     C
sysctl: unknown oid 'dev.cpu.1.temperature'
/tmp/1: line 72: [: : integer expression expected
CPU  1:     C
sysctl: unknown oid 'dev.cpu.2.temperature'
/tmp/1: line 72: [: : integer expression expected
CPU  2:     C
sysctl: unknown oid 'dev.cpu.3.temperature'
/tmp/1: line 72: [: : integer expression expected
CPU  3:     C
sysctl: unknown oid 'dev.cpu.4.temperature'
/tmp/1: line 72: [: : integer expression expected
CPU  4:     C
sysctl: unknown oid 'dev.cpu.5.temperature'
/tmp/1: line 72: [: : integer expression expected
CPU  5:     C
sysctl: unknown oid 'dev.cpu.6.temperature'
/tmp/1: line 72: [: : integer expression expected
CPU  6:     C
sysctl: unknown oid 'dev.cpu.7.temperature'
/tmp/1: line 72: [: : integer expression expected
CPU  7:     C

=== DRIVES ===
  ada0:   21C [32.0GB] E074080539           SiliconMotion based SSDs (TS32GSSD370S)
  ada1:   18C [32.0GB] E074080508           SiliconMotion based SSDs (TS32GSSD370S)
  ada2:   23C [3.00TB] WD-WCC7K7ZLRND9      Western Digital Red (WDC WD30EFRX-68N32N0)
  ada3:   25C [3.00TB] WD-WCC7K2SPH75X      Western Digital Red (WDC WD30EFRX-68N32N0)
  ada4:   24C [3.00TB] WD-WCC4N1VH31F1      Western Digital Red (WDC WD30EFRX-68EUZN0)
  ada5:   24C [3.00TB] WD-WCC4N5RZNSPE      Western Digital Red (WDC WD30EFRX-68EUZN0)
  ada6:   23C [3.00TB] WD-WCC4N1NHX8A5      Western Digital Red (WDC WD30EFRX-68EUZN0)
  ada7:   25C [3.00TB] WD-WCC4N2AFXK14      Western Digital Red (WDC WD30EFRX-68EUZN0)
   da0:   27C [250GB]  S2R6NX0JB51225X      Samsung based SSDs (Samsung SSD 850 EVO 250GB)
   da1:   16C [32.0GB] E074080587           SiliconMotion based SSDs (TS32GSSD370S)
   da2:   16C [32.0GB] E074080531           SiliconMotion based SSDs (TS32GSSD370S)
   da3:   27C [250GB]  S2R6NX0JB50287W      Samsung based SSDs (Samsung SSD 850 EVO 250GB)

root@freenas:~ # perl get-system-temps.pl
/tmp/2: line 22: use: command not found
/tmp/2: line 23: use: command not found
/tmp/2: line 27: syntax error near unexpected token `('
/tmp/2: line 27: `my $hostname = qx(hostname);'

root@freenas:/tmp # bash save_config_enc.sh
Backup configuration database file: /mnt/HDD/Media/Config/FreeNAS/freenas-FreeNAS-11.1-U1-f7e246b8f-20180131184136.db
sqlite3 status: [ok]
a freenas-FreeNAS-11.1-U1-f7e246b8f-20180131184136.db
tar status: [0]
options are
-in <file>     input file
-out <file>    output file
-pass <arg>    pass phrase source
-e             encrypt
-d             decrypt
-a/-base64     base64 encode/decode, depending on encryption flag
-k             passphrase is the next argument
-kfile         passphrase is the first line of the file argument
-md            the next argument is the md to use to create a key
                 from a passphrase.  One of md2, md5, sha or sha1
-S             salt in hex is the next argument
-K/-iv         key/iv in hex is the next argument
-[pP]          print the iv/key (then exit if -P)
-bufsize <n>   buffer size
-nopad         disable standard block padding
-engine e      use engine e, possibly a hardware device.
Cipher Types
-aes-128-cbc               -aes-128-cbc-hmac-sha1     -aes-128-cbc-hmac-sha256  
-aes-128-ccm               -aes-128-cfb               -aes-128-cfb1             
-aes-128-cfb8              -aes-128-ctr               -aes-128-ecb              
-aes-128-gcm               -aes-128-ofb               -aes-128-xts              
-aes-192-cbc               -aes-192-ccm               -aes-192-cfb              
-aes-192-cfb1              -aes-192-cfb8              -aes-192-ctr              
-aes-192-ecb               -aes-192-gcm               -aes-192-ofb              
-aes-256-cbc               -aes-256-cbc-hmac-sha1     -aes-256-cbc-hmac-sha256  
-aes-256-ccm               -aes-256-cfb               -aes-256-cfb1             
-aes-256-cfb8              -aes-256-ctr               -aes-256-ecb              
-aes-256-gcm               -aes-256-ofb               -aes-256-xts              
-aes128                    -aes192                    -aes256                   
-bf                        -bf-cbc                    -bf-cfb                   
-bf-ecb                    -bf-ofb                    -blowfish                 
-camellia-128-cbc          -camellia-128-cfb          -camellia-128-cfb1        
-camellia-128-cfb8         -camellia-128-ecb          -camellia-128-ofb         
-camellia-192-cbc          -camellia-192-cfb          -camellia-192-cfb1        
-camellia-192-cfb8         -camellia-192-ecb          -camellia-192-ofb         
-camellia-256-cbc          -camellia-256-cfb          -camellia-256-cfb1        
-camellia-256-cfb8         -camellia-256-ecb          -camellia-256-ofb         
-camellia128               -camellia192               -camellia256              
-cast                      -cast-cbc                  -cast5-cbc                
-cast5-cfb                 -cast5-ecb                 -cast5-ofb                
-des                       -des-cbc                   -des-cfb                  
-des-cfb1                  -des-cfb8                  -des-ecb                  
-des-ede                   -des-ede-cbc               -des-ede-cfb              
-des-ede-ofb               -des-ede3                  -des-ede3-cbc             
-des-ede3-cfb              -des-ede3-cfb1             -des-ede3-cfb8            
-des-ede3-ofb              -des-ofb                   -des3                     
-desx                      -desx-cbc                  -id-aes128-CCM            
-id-aes128-GCM             -id-aes128-wrap            -id-aes192-CCM            
-id-aes192-GCM             -id-aes192-wrap            -id-aes256-CCM            
-id-aes256-GCM             -id-aes256-wrap            -id-smime-alg-CMS3DESwrap 
-idea                      -idea-cbc                  -idea-cfb                 
-idea-ecb                  -idea-ofb                  -rc2                      
-rc2-40-cbc                -rc2-64-cbc                -rc2-cbc                  
-rc2-cfb                   -rc2-ecb                   -rc2-ofb                  
-rc4                       -rc4-40                    -rc4-hmac-md5             
-rc5                       -rc5-cbc                   -rc5-cfb                  
-rc5-ecb                   -rc5-ofb                   -seed                     
-seed-cbc                  -seed-cfb                  -seed-ecb                 
-seed-ofb                  
config.sh: line 106: file:/root/config_passphrase: No such file or directory
openssl status: [127]

File is created actually the config_passphrase with perm 0600
Seems to hang at the last output line...

root@freenas:/tmp # bash save_config.sh 
Backup FreeNAS configuration database file: /mnt/HDD/Media/Config/FreeNAS/freenas-FreeNAS-11.1-U1-f7e246b8f-20180131184430.db

Seems to hang here too..

root@freenas:/tmp # bash smart_report.sh 
root@freenas:/tmp # bash -x smart_report.sh
+ [email protected]
++ hostname -s
++ tr '[:lower:]' '[:upper:]'
+ freenashost=FREENAS
+ smartctl=/usr/local/sbin/smartctl
+ logfile=/tmp/smart_report.tmp
+ subject='SMART Status Report for FREENAS'
+ tempWarn=40
+ tempCrit=45
+ sectorsCrit=10
+ testAgeWarn=1
+ warnSymbol='?'
+ critSymbol='!'
+ drives=
+ get_smart_drives drives
++ /usr/local/sbin/smartctl --scan
++ grep dev
++ awk '{print $1}'
++ sed -e 's/\/dev\///'
++ tr '\n' ' '
+ gs_drives='ada0 ada1 ada2 ada3 ada4 ada5 ada6 ada7 da0 da1 da2 da3 '
+ gs_smartdrives=
+ for gs_drive in $gs_drives
++ /usr/local/sbin/smartctl -i /dev/ada0
++ grep 'SMART support is: Enabled'
++ awk '{print $4}'
+ gs_smart_flag=Enabled
+ '[' Enabled = Enabled ']'
+ gs_smartdrives=' ada0'
+ for gs_drive in $gs_drives
++ /usr/local/sbin/smartctl -i /dev/ada1
++ grep 'SMART support is: Enabled'
++ awk '{print $4}'
+ gs_smart_flag=Enabled
+ '[' Enabled = Enabled ']'
+ gs_smartdrives=' ada0 ada1'
+ for gs_drive in $gs_drives
++ /usr/local/sbin/smartctl -i /dev/ada2
++ grep 'SMART support is: Enabled'
++ awk '{print $4}'
+ gs_smart_flag=Enabled
+ '[' Enabled = Enabled ']'
+ gs_smartdrives=' ada0 ada1 ada2'
+ for gs_drive in $gs_drives
++ /usr/local/sbin/smartctl -i /dev/ada3
++ grep 'SMART support is: Enabled'
++ awk '{print $4}'
+ gs_smart_flag=Enabled
+ '[' Enabled = Enabled ']'
+ gs_smartdrives=' ada0 ada1 ada2 ada3'
+ for gs_drive in $gs_drives
++ /usr/local/sbin/smartctl -i /dev/ada4
++ grep 'SMART support is: Enabled'
++ awk '{print $4}'
+ gs_smart_flag=Enabled
+ '[' Enabled = Enabled ']'
+ gs_smartdrives=' ada0 ada1 ada2 ada3 ada4'
+ for gs_drive in $gs_drives
++ /usr/local/sbin/smartctl -i /dev/ada5
++ grep 'SMART support is: Enabled'
++ awk '{print $4}'
+ gs_smart_flag=Enabled
+ '[' Enabled = Enabled ']'
+ gs_smartdrives=' ada0 ada1 ada2 ada3 ada4 ada5'
+ for gs_drive in $gs_drives
++ /usr/local/sbin/smartctl -i /dev/ada6
++ grep 'SMART support is: Enabled'
++ awk '{print $4}'
+ gs_smart_flag=Enabled
+ '[' Enabled = Enabled ']'
+ gs_smartdrives=' ada0 ada1 ada2 ada3 ada4 ada5 ada6'
+ for gs_drive in $gs_drives
++ /usr/local/sbin/smartctl -i /dev/ada7
++ grep 'SMART support is: Enabled'
++ awk '{print $4}'
+ gs_smart_flag=Enabled
+ '[' Enabled = Enabled ']'
+ gs_smartdrives=' ada0 ada1 ada2 ada3 ada4 ada5 ada6 ada7'
+ for gs_drive in $gs_drives
++ /usr/local/sbin/smartctl -i /dev/da0
++ grep 'SMART support is: Enabled'
++ awk '{print $4}'
+ gs_smart_flag=Enabled
+ '[' Enabled = Enabled ']'
+ gs_smartdrives=' ada0 ada1 ada2 ada3 ada4 ada5 ada6 ada7 da0'
+ for gs_drive in $gs_drives
++ /usr/local/sbin/smartctl -i /dev/da1
++ grep 'SMART support is: Enabled'
++ awk '{print $4}'
+ gs_smart_flag=Enabled
+ '[' Enabled = Enabled ']'
+ gs_smartdrives=' ada0 ada1 ada2 ada3 ada4 ada5 ada6 ada7 da0 da1'
+ for gs_drive in $gs_drives
++ /usr/local/sbin/smartctl -i /dev/da2
++ grep 'SMART support is: Enabled'
++ awk '{print $4}'
+ gs_smart_flag=Enabled
+ '[' Enabled = Enabled ']'
+ gs_smartdrives=' ada0 ada1 ada2 ada3 ada4 ada5 ada6 ada7 da0 da1 da2'
+ for gs_drive in $gs_drives
++ /usr/local/sbin/smartctl -i /dev/da3
++ grep 'SMART support is: Enabled'
++ awk '{print $4}'
+ gs_smart_flag=Enabled
+ '[' Enabled = Enabled ']'
+ gs_smartdrives=' ada0 ada1 ada2 ada3 ada4 ada5 ada6 ada7 da0 da1 da2 da3'
+ eval 'drives=$gs_smartdrives'
++ drives=' ada0 ada1 ada2 ada3 ada4 ada5 ada6 ada7 da0 da1 da2 da3'
+ echo 'To: [email protected]'
+ echo 'Subject: SMART Status Report for FREENAS'
+ echo 'Content-Type: text/html'
+ echo 'MIME-Version: 1.0'
+ printf '\r\n'
+ echo '<pre style="font-size:14px">'
+ echo '########## SMART status report summary for all drives on server FREENAS ##########'
+ echo ''
+ echo +------+------------------+----+-----+-----+-----+-------+-------+--------+------+----------+------+-------+----+
+ echo '|Device|Serial            |Temp|Power|Start|Spin |ReAlloc|Current|Offline |Seek  |Total     |High  |Command|Last|'
+ echo '|      |Number            |    |On   |Stop |Retry|Sectors|Pending|Uncorrec|Errors|Seeks     |Fly   |Timeout|Test|'
+ echo '|      |                  |    |Hours|Count|Count|       |Sectors|Sectors |      |          |Writes|Count  |Age |'
+ echo +------+------------------+----+-----+-----+-----+-------+-------+--------+------+----------+------+-------+----+
+ for drive in $drives
++ /usr/local/sbin/smartctl -l selftest /dev/ada0
++ grep '# 1'
++ awk '{print $9}'
+ lastTestHours=0
+ /usr/local/sbin/smartctl -A -i -v 7,hex48 /dev/ada0
+ awk -v device=ada0 -v tempWarn=40 -v tempCrit=45 -v sectorsCrit=10 -v testAgeWarn=1 -v 'warnSymbol=?' -v 'critSymbol=!' -v lastTestHours=0 '
  /Serial Number:/{serial=$3}
  /190 Airflow_Temperature/{temp=$10}
  /194 Temperature/{temp=$10}
  /Power_On_Hours/{split($10,a,"+");sub(/h/,"",a[1]);onHours=a[1];}
  /Start_Stop_Count/{startStop=$10}
  /Spin_Retry_Count/{spinRetry=$10}
  /Reallocated_Sector/{reAlloc=$10}
  /Current_Pending_Sector/{pending=$10}
  /Offline_Uncorrectable/{offlineUnc=$10}
  /Seek_Error_Rate/{seekErrors=("0x" substr($10,3,4));totalSeeks=("0x" substr($10,7))}
  /High_Fly_Writes/{hiFlyWr=$10}
  /Command_Timeout/{cmdTimeout=$10}
  END {
      testAge=sprintf("%.0f", (onHours - lastTestHours) / 24);
      if (temp > tempCrit || reAlloc > sectorsCrit || pending > sectorsCrit || offlineUnc > sectorsCrit)
          device=device " " critSymbol;
      else if (temp > tempWarn || reAlloc > 0 || pending > 0 || offlineUnc > 0 || testAge > testAgeWarn)
          device=device " " warnSymbol;
      seekErrors=sprintf("%d", seekErrors);
      totalSeeks=sprintf("%d", totalSeeks);
      if (totalSeeks == "0") {
          seekErrors="N/A";
          totalSeeks="N/A";
      }
      if (hiFlyWr == "") hiFlyWr="N/A";
      if (cmdTimeout == "") cmdTimeout="N/A";
      printf "|%-6s|%-18s| %s |%5s|%5s|%5s|%7s|%7s|%8s|%6s|%10s|%6s|%7s|%4s|\n",
      device, serial, temp, onHours, startStop, spinRetry, reAlloc, pending, offlineUnc,
      seekErrors, totalSeeks, hiFlyWr, cmdTimeout, testAge;
      }'
+ for drive in $drives
++ /usr/local/sbin/smartctl -l selftest /dev/ada1
++ grep '# 1'
++ awk '{print $9}'
+ lastTestHours=0
+ /usr/local/sbin/smartctl -A -i -v 7,hex48 /dev/ada1
+ awk -v device=ada1 -v tempWarn=40 -v tempCrit=45 -v sectorsCrit=10 -v testAgeWarn=1 -v 'warnSymbol=?' -v 'critSymbol=!' -v lastTestHours=0 '
  /Serial Number:/{serial=$3}
  /190 Airflow_Temperature/{temp=$10}
  /194 Temperature/{temp=$10}
  /Power_On_Hours/{split($10,a,"+");sub(/h/,"",a[1]);onHours=a[1];}
  /Start_Stop_Count/{startStop=$10}
  /Spin_Retry_Count/{spinRetry=$10}
  /Reallocated_Sector/{reAlloc=$10}
  /Current_Pending_Sector/{pending=$10}
  /Offline_Uncorrectable/{offlineUnc=$10}
  /Seek_Error_Rate/{seekErrors=("0x" substr($10,3,4));totalSeeks=("0x" substr($10,7))}
  /High_Fly_Writes/{hiFlyWr=$10}
  /Command_Timeout/{cmdTimeout=$10}
  END {
      testAge=sprintf("%.0f", (onHours - lastTestHours) / 24);
      if (temp > tempCrit || reAlloc > sectorsCrit || pending > sectorsCrit || offlineUnc > sectorsCrit)
          device=device " " critSymbol;
      else if (temp > tempWarn || reAlloc > 0 || pending > 0 || offlineUnc > 0 || testAge > testAgeWarn)
          device=device " " warnSymbol;
      seekErrors=sprintf("%d", seekErrors);
      totalSeeks=sprintf("%d", totalSeeks);
      if (totalSeeks == "0") {
          seekErrors="N/A";
          totalSeeks="N/A";
      }
      if (hiFlyWr == "") hiFlyWr="N/A";
      if (cmdTimeout == "") cmdTimeout="N/A";
      printf "|%-6s|%-18s| %s |%5s|%5s|%5s|%7s|%7s|%8s|%6s|%10s|%6s|%7s|%4s|\n",
      device, serial, temp, onHours, startStop, spinRetry, reAlloc, pending, offlineUnc,
      seekErrors, totalSeeks, hiFlyWr, cmdTimeout, testAge;
      }'
+ for drive in $drives
++ /usr/local/sbin/smartctl -l selftest /dev/ada2
++ grep '# 1'
++ awk '{print $9}'
+ lastTestHours=28
+ /usr/local/sbin/smartctl -A -i -v 7,hex48 /dev/ada2
+ awk -v device=ada2 -v tempWarn=40 -v tempCrit=45 -v sectorsCrit=10 -v testAgeWarn=1 -v 'warnSymbol=?' -v 'critSymbol=!' -v lastTestHours=28 '
  /Serial Number:/{serial=$3}
  /190 Airflow_Temperature/{temp=$10}
  /194 Temperature/{temp=$10}
  /Power_On_Hours/{split($10,a,"+");sub(/h/,"",a[1]);onHours=a[1];}
  /Start_Stop_Count/{startStop=$10}
  /Spin_Retry_Count/{spinRetry=$10}
  /Reallocated_Sector/{reAlloc=$10}
  /Current_Pending_Sector/{pending=$10}
  /Offline_Uncorrectable/{offlineUnc=$10}
  /Seek_Error_Rate/{seekErrors=("0x" substr($10,3,4));totalSeeks=("0x" substr($10,7))}
  /High_Fly_Writes/{hiFlyWr=$10}
  /Command_Timeout/{cmdTimeout=$10}
  END {
      testAge=sprintf("%.0f", (onHours - lastTestHours) / 24);
      if (temp > tempCrit || reAlloc > sectorsCrit || pending > sectorsCrit || offlineUnc > sectorsCrit)
          device=device " " critSymbol;
      else if (temp > tempWarn || reAlloc > 0 || pending > 0 || offlineUnc > 0 || testAge > testAgeWarn)
          device=device " " warnSymbol;
      seekErrors=sprintf("%d", seekErrors);
      totalSeeks=sprintf("%d", totalSeeks);
      if (totalSeeks == "0") {
          seekErrors="N/A";
          totalSeeks="N/A";
      }
      if (hiFlyWr == "") hiFlyWr="N/A";
      if (cmdTimeout == "") cmdTimeout="N/A";
      printf "|%-6s|%-18s| %s |%5s|%5s|%5s|%7s|%7s|%8s|%6s|%10s|%6s|%7s|%4s|\n",
      device, serial, temp, onHours, startStop, spinRetry, reAlloc, pending, offlineUnc,
      seekErrors, totalSeeks, hiFlyWr, cmdTimeout, testAge;
      }'
+ for drive in $drives
++ /usr/local/sbin/smartctl -l selftest /dev/ada3
++ grep '# 1'
++ awk '{print $9}'
+ lastTestHours=23
+ /usr/local/sbin/smartctl -A -i -v 7,hex48 /dev/ada3
+ awk -v device=ada3 -v tempWarn=40 -v tempCrit=45 -v sectorsCrit=10 -v testAgeWarn=1 -v 'warnSymbol=?' -v 'critSymbol=!' -v lastTestHours=23 '
  /Serial Number:/{serial=$3}
  /190 Airflow_Temperature/{temp=$10}
  /194 Temperature/{temp=$10}
  /Power_On_Hours/{split($10,a,"+");sub(/h/,"",a[1]);onHours=a[1];}
  /Start_Stop_Count/{startStop=$10}
  /Spin_Retry_Count/{spinRetry=$10}
  /Reallocated_Sector/{reAlloc=$10}
  /Current_Pending_Sector/{pending=$10}
  /Offline_Uncorrectable/{offlineUnc=$10}
  /Seek_Error_Rate/{seekErrors=("0x" substr($10,3,4));totalSeeks=("0x" substr($10,7))}
  /High_Fly_Writes/{hiFlyWr=$10}
  /Command_Timeout/{cmdTimeout=$10}
  END {
      testAge=sprintf("%.0f", (onHours - lastTestHours) / 24);
      if (temp > tempCrit || reAlloc > sectorsCrit || pending > sectorsCrit || offlineUnc > sectorsCrit)
          device=device " " critSymbol;
      else if (temp > tempWarn || reAlloc > 0 || pending > 0 || offlineUnc > 0 || testAge > testAgeWarn)
          device=device " " warnSymbol;
      seekErrors=sprintf("%d", seekErrors);
      totalSeeks=sprintf("%d", totalSeeks);
      if (totalSeeks == "0") {
          seekErrors="N/A";
          totalSeeks="N/A";
      }
      if (hiFlyWr == "") hiFlyWr="N/A";
      if (cmdTimeout == "") cmdTimeout="N/A";
      printf "|%-6s|%-18s| %s |%5s|%5s|%5s|%7s|%7s|%8s|%6s|%10s|%6s|%7s|%4s|\n",
      device, serial, temp, onHours, startStop, spinRetry, reAlloc, pending, offlineUnc,
      seekErrors, totalSeeks, hiFlyWr, cmdTimeout, testAge;
      }'
+ for drive in $drives
++ /usr/local/sbin/smartctl -l selftest /dev/ada4
++ grep '# 1'
++ awk '{print $9}'
+ lastTestHours=32
+ /usr/local/sbin/smartctl -A -i -v 7,hex48 /dev/ada4
+ awk -v device=ada4 -v tempWarn=40 -v tempCrit=45 -v sectorsCrit=10 -v testAgeWarn=1 -v 'warnSymbol=?' -v 'critSymbol=!' -v lastTestHours=32 '
  /Serial Number:/{serial=$3}
  /190 Airflow_Temperature/{temp=$10}
  /194 Temperature/{temp=$10}
  /Power_On_Hours/{split($10,a,"+");sub(/h/,"",a[1]);onHours=a[1];}
  /Start_Stop_Count/{startStop=$10}
  /Spin_Retry_Count/{spinRetry=$10}
  /Reallocated_Sector/{reAlloc=$10}
  /Current_Pending_Sector/{pending=$10}
  /Offline_Uncorrectable/{offlineUnc=$10}
  /Seek_Error_Rate/{seekErrors=("0x" substr($10,3,4));totalSeeks=("0x" substr($10,7))}
  /High_Fly_Writes/{hiFlyWr=$10}
  /Command_Timeout/{cmdTimeout=$10}
  END {
      testAge=sprintf("%.0f", (onHours - lastTestHours) / 24);
      if (temp > tempCrit || reAlloc > sectorsCrit || pending > sectorsCrit || offlineUnc > sectorsCrit)
          device=device " " critSymbol;
      else if (temp > tempWarn || reAlloc > 0 || pending > 0 || offlineUnc > 0 || testAge > testAgeWarn)
          device=device " " warnSymbol;
      seekErrors=sprintf("%d", seekErrors);
      totalSeeks=sprintf("%d", totalSeeks);
      if (totalSeeks == "0") {
          seekErrors="N/A";
          totalSeeks="N/A";
      }
      if (hiFlyWr == "") hiFlyWr="N/A";
      if (cmdTimeout == "") cmdTimeout="N/A";
      printf "|%-6s|%-18s| %s |%5s|%5s|%5s|%7s|%7s|%8s|%6s|%10s|%6s|%7s|%4s|\n",
      device, serial, temp, onHours, startStop, spinRetry, reAlloc, pending, offlineUnc,
      seekErrors, totalSeeks, hiFlyWr, cmdTimeout, testAge;
      }'
+ for drive in $drives
++ /usr/local/sbin/smartctl -l selftest /dev/ada5
++ grep '# 1'
++ awk '{print $9}'
+ lastTestHours=22747
+ /usr/local/sbin/smartctl -A -i -v 7,hex48 /dev/ada5
+ awk -v device=ada5 -v tempWarn=40 -v tempCrit=45 -v sectorsCrit=10 -v testAgeWarn=1 -v 'warnSymbol=?' -v 'critSymbol=!' -v lastTestHours=22747 '
  /Serial Number:/{serial=$3}
  /190 Airflow_Temperature/{temp=$10}
  /194 Temperature/{temp=$10}
  /Power_On_Hours/{split($10,a,"+");sub(/h/,"",a[1]);onHours=a[1];}
  /Start_Stop_Count/{startStop=$10}
  /Spin_Retry_Count/{spinRetry=$10}
  /Reallocated_Sector/{reAlloc=$10}
  /Current_Pending_Sector/{pending=$10}
  /Offline_Uncorrectable/{offlineUnc=$10}
  /Seek_Error_Rate/{seekErrors=("0x" substr($10,3,4));totalSeeks=("0x" substr($10,7))}
  /High_Fly_Writes/{hiFlyWr=$10}
  /Command_Timeout/{cmdTimeout=$10}
  END {
      testAge=sprintf("%.0f", (onHours - lastTestHours) / 24);
      if (temp > tempCrit || reAlloc > sectorsCrit || pending > sectorsCrit || offlineUnc > sectorsCrit)
          device=device " " critSymbol;
      else if (temp > tempWarn || reAlloc > 0 || pending > 0 || offlineUnc > 0 || testAge > testAgeWarn)
          device=device " " warnSymbol;
      seekErrors=sprintf("%d", seekErrors);
      totalSeeks=sprintf("%d", totalSeeks);
      if (totalSeeks == "0") {
          seekErrors="N/A";
          totalSeeks="N/A";
      }
      if (hiFlyWr == "") hiFlyWr="N/A";
      if (cmdTimeout == "") cmdTimeout="N/A";
      printf "|%-6s|%-18s| %s |%5s|%5s|%5s|%7s|%7s|%8s|%6s|%10s|%6s|%7s|%4s|\n",
      device, serial, temp, onHours, startStop, spinRetry, reAlloc, pending, offlineUnc,
      seekErrors, totalSeeks, hiFlyWr, cmdTimeout, testAge;
      }'
+ for drive in $drives
++ /usr/local/sbin/smartctl -l selftest /dev/ada6
++ grep '# 1'
++ awk '{print $9}'
+ lastTestHours=32
+ /usr/local/sbin/smartctl -A -i -v 7,hex48 /dev/ada6
+ awk -v device=ada6 -v tempWarn=40 -v tempCrit=45 -v sectorsCrit=10 -v testAgeWarn=1 -v 'warnSymbol=?' -v 'critSymbol=!' -v lastTestHours=32 '
  /Serial Number:/{serial=$3}
  /190 Airflow_Temperature/{temp=$10}
  /194 Temperature/{temp=$10}
  /Power_On_Hours/{split($10,a,"+");sub(/h/,"",a[1]);onHours=a[1];}
  /Start_Stop_Count/{startStop=$10}
  /Spin_Retry_Count/{spinRetry=$10}
  /Reallocated_Sector/{reAlloc=$10}
  /Current_Pending_Sector/{pending=$10}
  /Offline_Uncorrectable/{offlineUnc=$10}
  /Seek_Error_Rate/{seekErrors=("0x" substr($10,3,4));totalSeeks=("0x" substr($10,7))}
  /High_Fly_Writes/{hiFlyWr=$10}
  /Command_Timeout/{cmdTimeout=$10}
  END {
      testAge=sprintf("%.0f", (onHours - lastTestHours) / 24);
      if (temp > tempCrit || reAlloc > sectorsCrit || pending > sectorsCrit || offlineUnc > sectorsCrit)
          device=device " " critSymbol;
      else if (temp > tempWarn || reAlloc > 0 || pending > 0 || offlineUnc > 0 || testAge > testAgeWarn)
          device=device " " warnSymbol;
      seekErrors=sprintf("%d", seekErrors);
      totalSeeks=sprintf("%d", totalSeeks);
      if (totalSeeks == "0") {
          seekErrors="N/A";
          totalSeeks="N/A";
      }
      if (hiFlyWr == "") hiFlyWr="N/A";
      if (cmdTimeout == "") cmdTimeout="N/A";
      printf "|%-6s|%-18s| %s |%5s|%5s|%5s|%7s|%7s|%8s|%6s|%10s|%6s|%7s|%4s|\n",
      device, serial, temp, onHours, startStop, spinRetry, reAlloc, pending, offlineUnc,
      seekErrors, totalSeeks, hiFlyWr, cmdTimeout, testAge;
      }'
+ for drive in $drives
++ /usr/local/sbin/smartctl -l selftest /dev/ada7
++ grep '# 1'
++ awk '{print $9}'
+ lastTestHours=32
+ /usr/local/sbin/smartctl -A -i -v 7,hex48 /dev/ada7
+ awk -v device=ada7 -v tempWarn=40 -v tempCrit=45 -v sectorsCrit=10 -v testAgeWarn=1 -v 'warnSymbol=?' -v 'critSymbol=!' -v lastTestHours=32 '
  /Serial Number:/{serial=$3}
  /190 Airflow_Temperature/{temp=$10}
  /194 Temperature/{temp=$10}
  /Power_On_Hours/{split($10,a,"+");sub(/h/,"",a[1]);onHours=a[1];}
  /Start_Stop_Count/{startStop=$10}
  /Spin_Retry_Count/{spinRetry=$10}
  /Reallocated_Sector/{reAlloc=$10}
  /Current_Pending_Sector/{pending=$10}
  /Offline_Uncorrectable/{offlineUnc=$10}
  /Seek_Error_Rate/{seekErrors=("0x" substr($10,3,4));totalSeeks=("0x" substr($10,7))}
  /High_Fly_Writes/{hiFlyWr=$10}
  /Command_Timeout/{cmdTimeout=$10}
  END {
      testAge=sprintf("%.0f", (onHours - lastTestHours) / 24);
      if (temp > tempCrit || reAlloc > sectorsCrit || pending > sectorsCrit || offlineUnc > sectorsCrit)
          device=device " " critSymbol;
      else if (temp > tempWarn || reAlloc > 0 || pending > 0 || offlineUnc > 0 || testAge > testAgeWarn)
          device=device " " warnSymbol;
      seekErrors=sprintf("%d", seekErrors);
      totalSeeks=sprintf("%d", totalSeeks);
      if (totalSeeks == "0") {
          seekErrors="N/A";
          totalSeeks="N/A";
      }
      if (hiFlyWr == "") hiFlyWr="N/A";
      if (cmdTimeout == "") cmdTimeout="N/A";
      printf "|%-6s|%-18s| %s |%5s|%5s|%5s|%7s|%7s|%8s|%6s|%10s|%6s|%7s|%4s|\n",
      device, serial, temp, onHours, startStop, spinRetry, reAlloc, pending, offlineUnc,
      seekErrors, totalSeeks, hiFlyWr, cmdTimeout, testAge;
      }'
+ for drive in $drives
++ /usr/local/sbin/smartctl -l selftest /dev/da0
++ grep '# 1'
++ awk '{print $9}'
+ lastTestHours=-
+ /usr/local/sbin/smartctl -A -i -v 7,hex48 /dev/da0
+ awk -v device=da0 -v tempWarn=40 -v tempCrit=45 -v sectorsCrit=10 -v testAgeWarn=1 -v 'warnSymbol=?' -v 'critSymbol=!' -v lastTestHours=- '
  /Serial Number:/{serial=$3}
  /190 Airflow_Temperature/{temp=$10}
  /194 Temperature/{temp=$10}
  /Power_On_Hours/{split($10,a,"+");sub(/h/,"",a[1]);onHours=a[1];}
  /Start_Stop_Count/{startStop=$10}
  /Spin_Retry_Count/{spinRetry=$10}
  /Reallocated_Sector/{reAlloc=$10}
  /Current_Pending_Sector/{pending=$10}
  /Offline_Uncorrectable/{offlineUnc=$10}
  /Seek_Error_Rate/{seekErrors=("0x" substr($10,3,4));totalSeeks=("0x" substr($10,7))}
  /High_Fly_Writes/{hiFlyWr=$10}
  /Command_Timeout/{cmdTimeout=$10}
  END {
      testAge=sprintf("%.0f", (onHours - lastTestHours) / 24);
      if (temp > tempCrit || reAlloc > sectorsCrit || pending > sectorsCrit || offlineUnc > sectorsCrit)
          device=device " " critSymbol;
      else if (temp > tempWarn || reAlloc > 0 || pending > 0 || offlineUnc > 0 || testAge > testAgeWarn)
          device=device " " warnSymbol;
      seekErrors=sprintf("%d", seekErrors);
      totalSeeks=sprintf("%d", totalSeeks);
      if (totalSeeks == "0") {
          seekErrors="N/A";
          totalSeeks="N/A";
      }
      if (hiFlyWr == "") hiFlyWr="N/A";
      if (cmdTimeout == "") cmdTimeout="N/A";
      printf "|%-6s|%-18s| %s |%5s|%5s|%5s|%7s|%7s|%8s|%6s|%10s|%6s|%7s|%4s|\n",
      device, serial, temp, onHours, startStop, spinRetry, reAlloc, pending, offlineUnc,
      seekErrors, totalSeeks, hiFlyWr, cmdTimeout, testAge;
      }'
+ for drive in $drives
++ /usr/local/sbin/smartctl -l selftest /dev/da1
++ grep '# 1'
++ awk '{print $9}'
+ lastTestHours=0
+ /usr/local/sbin/smartctl -A -i -v 7,hex48 /dev/da1
+ awk -v device=da1 -v tempWarn=40 -v tempCrit=45 -v sectorsCrit=10 -v testAgeWarn=1 -v 'warnSymbol=?' -v 'critSymbol=!' -v lastTestHours=0 '
  /Serial Number:/{serial=$3}
  /190 Airflow_Temperature/{temp=$10}
  /194 Temperature/{temp=$10}
  /Power_On_Hours/{split($10,a,"+");sub(/h/,"",a[1]);onHours=a[1];}
  /Start_Stop_Count/{startStop=$10}
  /Spin_Retry_Count/{spinRetry=$10}
  /Reallocated_Sector/{reAlloc=$10}
  /Current_Pending_Sector/{pending=$10}
  /Offline_Uncorrectable/{offlineUnc=$10}
  /Seek_Error_Rate/{seekErrors=("0x" substr($10,3,4));totalSeeks=("0x" substr($10,7))}
  /High_Fly_Writes/{hiFlyWr=$10}
  /Command_Timeout/{cmdTimeout=$10}
  END {
      testAge=sprintf("%.0f", (onHours - lastTestHours) / 24);
      if (temp > tempCrit || reAlloc > sectorsCrit || pending > sectorsCrit || offlineUnc > sectorsCrit)
          device=device " " critSymbol;
      else if (temp > tempWarn || reAlloc > 0 || pending > 0 || offlineUnc > 0 || testAge > testAgeWarn)
          device=device " " warnSymbol;
      seekErrors=sprintf("%d", seekErrors);
      totalSeeks=sprintf("%d", totalSeeks);
      if (totalSeeks == "0") {
          seekErrors="N/A";
          totalSeeks="N/A";
      }
      if (hiFlyWr == "") hiFlyWr="N/A";
      if (cmdTimeout == "") cmdTimeout="N/A";
      printf "|%-6s|%-18s| %s |%5s|%5s|%5s|%7s|%7s|%8s|%6s|%10s|%6s|%7s|%4s|\n",
      device, serial, temp, onHours, startStop, spinRetry, reAlloc, pending, offlineUnc,
      seekErrors, totalSeeks, hiFlyWr, cmdTimeout, testAge;
      }'
+ for drive in $drives
++ /usr/local/sbin/smartctl -l selftest /dev/da2
++ grep '# 1'
++ awk '{print $9}'
+ lastTestHours=0
+ /usr/local/sbin/smartctl -A -i -v 7,hex48 /dev/da2
+ awk -v device=da2 -v tempWarn=40 -v tempCrit=45 -v sectorsCrit=10 -v testAgeWarn=1 -v 'warnSymbol=?' -v 'critSymbol=!' -v lastTestHours=0 '
  /Serial Number:/{serial=$3}
  /190 Airflow_Temperature/{temp=$10}
  /194 Temperature/{temp=$10}
  /Power_On_Hours/{split($10,a,"+");sub(/h/,"",a[1]);onHours=a[1];}
  /Start_Stop_Count/{startStop=$10}
  /Spin_Retry_Count/{spinRetry=$10}
  /Reallocated_Sector/{reAlloc=$10}
  /Current_Pending_Sector/{pending=$10}
  /Offline_Uncorrectable/{offlineUnc=$10}
  /Seek_Error_Rate/{seekErrors=("0x" substr($10,3,4));totalSeeks=("0x" substr($10,7))}
  /High_Fly_Writes/{hiFlyWr=$10}
  /Command_Timeout/{cmdTimeout=$10}
  END {
      testAge=sprintf("%.0f", (onHours - lastTestHours) / 24);
      if (temp > tempCrit || reAlloc > sectorsCrit || pending > sectorsCrit || offlineUnc > sectorsCrit)
          device=device " " critSymbol;
      else if (temp > tempWarn || reAlloc > 0 || pending > 0 || offlineUnc > 0 || testAge > testAgeWarn)
          device=device " " warnSymbol;
      seekErrors=sprintf("%d", seekErrors);
      totalSeeks=sprintf("%d", totalSeeks);
      if (totalSeeks == "0") {
          seekErrors="N/A";
          totalSeeks="N/A";
      }
      if (hiFlyWr == "") hiFlyWr="N/A";
      if (cmdTimeout == "") cmdTimeout="N/A";
      printf "|%-6s|%-18s| %s |%5s|%5s|%5s|%7s|%7s|%8s|%6s|%10s|%6s|%7s|%4s|\n",
      device, serial, temp, onHours, startStop, spinRetry, reAlloc, pending, offlineUnc,
      seekErrors, totalSeeks, hiFlyWr, cmdTimeout, testAge;
      }'
+ for drive in $drives
++ /usr/local/sbin/smartctl -l selftest /dev/da3
++ grep '# 1'
++ awk '{print $9}'
+ lastTestHours=
+ /usr/local/sbin/smartctl -A -i -v 7,hex48 /dev/da3
+ awk -v device=da3 -v tempWarn=40 -v tempCrit=45 -v sectorsCrit=10 -v testAgeWarn=1 -v 'warnSymbol=?' -v 'critSymbol=!' -v lastTestHours= '
  /Serial Number:/{serial=$3}
  /190 Airflow_Temperature/{temp=$10}
  /194 Temperature/{temp=$10}
  /Power_On_Hours/{split($10,a,"+");sub(/h/,"",a[1]);onHours=a[1];}
  /Start_Stop_Count/{startStop=$10}
  /Spin_Retry_Count/{spinRetry=$10}
  /Reallocated_Sector/{reAlloc=$10}
  /Current_Pending_Sector/{pending=$10}
  /Offline_Uncorrectable/{offlineUnc=$10}
  /Seek_Error_Rate/{seekErrors=("0x" substr($10,3,4));totalSeeks=("0x" substr($10,7))}
  /High_Fly_Writes/{hiFlyWr=$10}
  /Command_Timeout/{cmdTimeout=$10}
  END {
      testAge=sprintf("%.0f", (onHours - lastTestHours) / 24);
      if (temp > tempCrit || reAlloc > sectorsCrit || pending > sectorsCrit || offlineUnc > sectorsCrit)
          device=device " " critSymbol;
      else if (temp > tempWarn || reAlloc > 0 || pending > 0 || offlineUnc > 0 || testAge > testAgeWarn)
          device=device " " warnSymbol;
      seekErrors=sprintf("%d", seekErrors);
      totalSeeks=sprintf("%d", totalSeeks);
      if (totalSeeks == "0") {
          seekErrors="N/A";
          totalSeeks="N/A";
      }
      if (hiFlyWr == "") hiFlyWr="N/A";
      if (cmdTimeout == "") cmdTimeout="N/A";
      printf "|%-6s|%-18s| %s |%5s|%5s|%5s|%7s|%7s|%8s|%6s|%10s|%6s|%7s|%4s|\n",
      device, serial, temp, onHours, startStop, spinRetry, reAlloc, pending, offlineUnc,
      seekErrors, totalSeeks, hiFlyWr, cmdTimeout, testAge;
      }'
+ echo +------+------------------+----+-----+-----+-----+-------+-------+--------+------+----------+------+-------+----+
+ for drive in $drives
++ /usr/local/sbin/smartctl -i /dev/ada0
++ grep 'Model Family'
++ awk '{print $3, $4, $5}'
+ brand='SiliconMotion based SSDs'
+ '[' -z 'SiliconMotion based SSDs' ']'
++ /usr/local/sbin/smartctl -i /dev/ada0
++ grep 'Serial Number'
++ awk '{print $3}'
+ serial=E074080539
+ echo ''
+ echo '########## SMART status report for ada0 drive (SiliconMotion based SSDs: E074080539) ##########'
+ /usr/local/sbin/smartctl -n never -H -A -l error /dev/ada0
+ /usr/local/sbin/smartctl -n never -l selftest /dev/ada0
+ grep '# 1 \|Num'
+ cut -c6-
+ for drive in $drives
++ /usr/local/sbin/smartctl -i /dev/ada1
++ grep 'Model Family'
++ awk '{print $3, $4, $5}'
+ brand='SiliconMotion based SSDs'
+ '[' -z 'SiliconMotion based SSDs' ']'
++ /usr/local/sbin/smartctl -i /dev/ada1
++ grep 'Serial Number'
++ awk '{print $3}'
+ serial=E074080508
+ echo ''
+ echo '########## SMART status report for ada1 drive (SiliconMotion based SSDs: E074080508) ##########'
+ /usr/local/sbin/smartctl -n never -H -A -l error /dev/ada1
+ /usr/local/sbin/smartctl -n never -l selftest /dev/ada1
+ grep '# 1 \|Num'
+ cut -c6-
+ for drive in $drives
++ /usr/local/sbin/smartctl -i /dev/ada2
++ grep 'Model Family'
++ awk '{print $3, $4, $5}'
+ brand='Western Digital Red'
+ '[' -z 'Western Digital Red' ']'
++ /usr/local/sbin/smartctl -i /dev/ada2
++ grep 'Serial Number'
++ awk '{print $3}'
+ serial=WD-WCC7K7ZLRND9
+ echo ''
+ echo '########## SMART status report for ada2 drive (Western Digital Red: WD-WCC7K7ZLRND9) ##########'
+ /usr/local/sbin/smartctl -n never -H -A -l error /dev/ada2
+ /usr/local/sbin/smartctl -n never -l selftest /dev/ada2
+ grep '# 1 \|Num'
+ cut -c6-
+ for drive in $drives
++ /usr/local/sbin/smartctl -i /dev/ada3
++ grep 'Model Family'
++ awk '{print $3, $4, $5}'
+ brand='Western Digital Red'
+ '[' -z 'Western Digital Red' ']'
++ /usr/local/sbin/smartctl -i /dev/ada3
++ grep 'Serial Number'
++ awk '{print $3}'
+ serial=WD-WCC7K2SPH75X
+ echo ''
+ echo '########## SMART status report for ada3 drive (Western Digital Red: WD-WCC7K2SPH75X) ##########'
+ /usr/local/sbin/smartctl -n never -H -A -l error /dev/ada3
+ /usr/local/sbin/smartctl -n never -l selftest /dev/ada3
+ grep '# 1 \|Num'
+ cut -c6-
+ for drive in $drives
++ /usr/local/sbin/smartctl -i /dev/ada4
++ grep 'Model Family'
++ awk '{print $3, $4, $5}'
+ brand='Western Digital Red'
+ '[' -z 'Western Digital Red' ']'
++ /usr/local/sbin/smartctl -i /dev/ada4
++ grep 'Serial Number'
++ awk '{print $3}'
+ serial=WD-WCC4N1VH31F1
+ echo ''
+ echo '########## SMART status report for ada4 drive (Western Digital Red: WD-WCC4N1VH31F1) ##########'
+ /usr/local/sbin/smartctl -n never -H -A -l error /dev/ada4
+ /usr/local/sbin/smartctl -n never -l selftest /dev/ada4
+ grep '# 1 \|Num'
+ cut -c6-
+ for drive in $drives
++ /usr/local/sbin/smartctl -i /dev/ada5
++ grep 'Model Family'
++ awk '{print $3, $4, $5}'
+ brand='Western Digital Red'
+ '[' -z 'Western Digital Red' ']'
++ /usr/local/sbin/smartctl -i /dev/ada5
++ grep 'Serial Number'
++ awk '{print $3}'
+ serial=WD-WCC4N5RZNSPE
+ echo ''
+ echo '########## SMART status report for ada5 drive (Western Digital Red: WD-WCC4N5RZNSPE) ##########'
+ /usr/local/sbin/smartctl -n never -H -A -l error /dev/ada5
+ /usr/local/sbin/smartctl -n never -l selftest /dev/ada5
+ grep '# 1 \|Num'
+ cut -c6-
+ for drive in $drives
++ /usr/local/sbin/smartctl -i /dev/ada6
++ grep 'Model Family'
++ awk '{print $3, $4, $5}'
+ brand='Western Digital Red'
+ '[' -z 'Western Digital Red' ']'
++ /usr/local/sbin/smartctl -i /dev/ada6
++ grep 'Serial Number'
++ awk '{print $3}'
+ serial=WD-WCC4N1NHX8A5
+ echo ''
+ echo '########## SMART status report for ada6 drive (Western Digital Red: WD-WCC4N1NHX8A5) ##########'
+ /usr/local/sbin/smartctl -n never -H -A -l error /dev/ada6
+ /usr/local/sbin/smartctl -n never -l selftest /dev/ada6
+ grep '# 1 \|Num'
+ cut -c6-
+ for drive in $drives
++ /usr/local/sbin/smartctl -i /dev/ada7
++ grep 'Model Family'
++ awk '{print $3, $4, $5}'
+ brand='Western Digital Red'
+ '[' -z 'Western Digital Red' ']'
++ /usr/local/sbin/smartctl -i /dev/ada7
++ grep 'Serial Number'
++ awk '{print $3}'
+ serial=WD-WCC4N2AFXK14
+ echo ''
+ echo '########## SMART status report for ada7 drive (Western Digital Red: WD-WCC4N2AFXK14) ##########'
+ /usr/local/sbin/smartctl -n never -H -A -l error /dev/ada7
+ /usr/local/sbin/smartctl -n never -l selftest /dev/ada7
+ grep '# 1 \|Num'
+ cut -c6-
+ for drive in $drives
++ /usr/local/sbin/smartctl -i /dev/da0
++ grep 'Model Family'
++ awk '{print $3, $4, $5}'
+ brand='Samsung based SSDs'
+ '[' -z 'Samsung based SSDs' ']'
++ /usr/local/sbin/smartctl -i /dev/da0
++ grep 'Serial Number'
++ awk '{print $3}'
+ serial=S2R6NX0JB51225X
+ echo ''
+ echo '########## SMART status report for da0 drive (Samsung based SSDs: S2R6NX0JB51225X) ##########'
+ /usr/local/sbin/smartctl -n never -H -A -l error /dev/da0
+ /usr/local/sbin/smartctl -n never -l selftest /dev/da0
+ grep '# 1 \|Num'
+ cut -c6-
+ for drive in $drives
++ /usr/local/sbin/smartctl -i /dev/da1
++ grep 'Model Family'
++ awk '{print $3, $4, $5}'
+ brand='SiliconMotion based SSDs'
+ '[' -z 'SiliconMotion based SSDs' ']'
++ /usr/local/sbin/smartctl -i /dev/da1
++ grep 'Serial Number'
++ awk '{print $3}'
+ serial=E074080587
+ echo ''
+ echo '########## SMART status report for da1 drive (SiliconMotion based SSDs: E074080587) ##########'
+ /usr/local/sbin/smartctl -n never -H -A -l error /dev/da1
+ /usr/local/sbin/smartctl -n never -l selftest /dev/da1
+ grep '# 1 \|Num'
+ cut -c6-
+ for drive in $drives
++ /usr/local/sbin/smartctl -i /dev/da2
++ grep 'Model Family'
++ awk '{print $3, $4, $5}'
+ brand='SiliconMotion based SSDs'
+ '[' -z 'SiliconMotion based SSDs' ']'
++ /usr/local/sbin/smartctl -i /dev/da2
++ grep 'Serial Number'
++ awk '{print $3}'
+ serial=E074080531
+ echo ''
+ echo '########## SMART status report for da2 drive (SiliconMotion based SSDs: E074080531) ##########'
+ /usr/local/sbin/smartctl -n never -H -A -l error /dev/da2
+ /usr/local/sbin/smartctl -n never -l selftest /dev/da2
+ grep '# 1 \|Num'
+ cut -c6-
+ for drive in $drives
++ /usr/local/sbin/smartctl -i /dev/da3
++ grep 'Model Family'
++ awk '{print $3, $4, $5}'
+ brand='Samsung based SSDs'
+ '[' -z 'Samsung based SSDs' ']'
++ /usr/local/sbin/smartctl -i /dev/da3
++ grep 'Serial Number'
++ awk '{print $3}'
+ serial=S2R6NX0JB50287W
+ echo ''
+ echo '########## SMART status report for da3 drive (Samsung based SSDs: S2R6NX0JB50287W) ##########'
+ /usr/local/sbin/smartctl -n never -H -A -l error /dev/da3
+ /usr/local/sbin/smartctl -n never -l selftest /dev/da3
+ grep '# 1 \|Num'
+ cut -c6-
+ sed -i '' -e '/smartctl 6.*/d' /tmp/smart_report.tmp
+ sed -i '' -e '/smartctl 5.*/d' /tmp/smart_report.tmp
+ sed -i '' -e '/smartctl 4.*/d' /tmp/smart_report.tmp
+ sed -i '' -e /Copyright/d /tmp/smart_report.tmp
+ sed -i '' -e '/=== START OF READ/d' /tmp/smart_report.tmp
+ sed -i '' -e '/SMART Attributes Data/d' /tmp/smart_report.tmp
+ sed -i '' -e '/Vendor Specific SMART/d' /tmp/smart_report.tmp
+ sed -i '' -e '/SMART Error Log Version/d' /tmp/smart_report.tmp
+ echo '</pre>'
+ '[' -z [email protected] ']'
+ sendmail [email protected]
+ rm /tmp/smart_report.tmp

bash ups_report.sh

I do have a UPS! Before you ask :)

bash zpool_report.sh

The last 3 don't seem to send the actual email, i receive nothing. FreeNAS notifications do work.

smartctl requires /dev/{drive}

Hello, thanks for the work on this. The HDD temps doesn't work on my system (11.2-U7) due to smartctl requiring /dev/ in front of the drive ID

$ sudo smartctl -A da1
smartctl 6.6 2017-11-05 r4594 [FreeBSD 11.2-STABLE amd64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

da1: Unable to detect device type
Please specify device type with the -d option.

$ sudo smartctl -A /dev/da1 | grep Temp
194 Temperature_Celsius     0x0022   119   098   000    Old_age   Always       -       33

CPU temps aren't numeric from sysctl, but get-system-temps.pl expect them to be

Argument "37.0C" isn't numeric in numeric le (<=) at ./get-system-temps.pl line 152.
CPU 47: 37.0CC

save_config_enc.sh sends email as root

Hi, found that save_config_enc.sh sends as root which means email ends up in spam.

Deleted From: root on line 133 to below appears to have done the trick.

printf '%s\n' "To: ${email} Subject: ${subject} Mime-Version: 1.0 Content-Type: multipart/mixed; boundary=\"$mime_boundary\" --${mime_boundary} Content-Type: text/plain; charset=\"US-ASCII\" Content-Transfer-Encoding: 7bit Content-Disposition: inline ${savestatus} Server: ${freenashostname} Version: ${freenasversion} File: ${fnconfigdest} "

date parsing in zpool_report.sh fails if scrub takes longer than 24 hrs

I have one TrueNAS 12 server with an encrypted pool and a slow CPU which takes more than 24 hours to do a scrub.

zpool status | grep scan
scan: scrub repaired 0B in 1 days 11:56:46 with 0 errors on Wed Dec 9 06:07:04 2020

This results in:

Failed conversion of 9-on-Wed_Dec'' using format %Y-%b-%e_%H:%M:%S''
date: illegal time format
usage: date [-jnRu] [-d dst] [-r seconds|file] [-t west] [-v[+|-]val[ymwdHMS]]

get_hdd_temp.sh not working alright with Dell iDRAC 8

My server is a Dell R730xd with iDRAC 8 and the way the script reads CPU temperatures will perhaps work for a SuperMicro IPMI, but not for a iDRAC 8 setup.
iDRAC only has a "Temp" entry per CPU. Not even "CPU 0 TEMP", just Temp.

So, in my server, a dual socket with only one occupied, the script shows this when using ipmi:

=== CPU (0) ===
[: : bad number
CPU 1: [C]
[: : bad number
CPU 0: [C]

When run part of the line from the script it shows this:

$ doas ipmitool -I lanplus -H 10.42.10.10 -U root -f /root/ipmi_password sdr elist all | grep Temp
Inlet Temp | 04h | ok | 7.1 | 24 degrees C
Exhaust Temp | 01h | ok | 7.1 | 33 degrees C
Temp | 0Eh | ok | 3.1 | 43 degrees C
Temp | 0Fh | ns | 3.2 | Disabled

The second Temp line is the second socket which is not used in my server.

Oddly enough, the iDRAC Temp is higher then what sysctl shows, but I am guessing iDRAC shows the package temperature and not some combined/divided temperature.

$ sh get_hdd_temp.sh
=== CPU (16) ===
CPU 0: 37C
CPU 1: 38C
CPU 2: 38C
CPU 3: 38C
CPU 4: 37C
CPU 5: 37C
CPU 6: 36C
CPU 7: 36C
CPU 8: 37C
CPU 9: 38C
CPU 10: 36C
CPU 11: 37C
CPU 12: 38C
CPU 13: 39C
CPU 14: 36C
CPU 15: 36C

I checked with sysctl but there doesn't seem to be a package temperature available.

It would be nice to have iDRAC support.
I can deliver a full "sdr elist all" if wanted. It's rather long.

zpool_report.sh date error TrueNAS 12.0-U8.1

getting this error from zpool_report.sh on TrueNAS 12.0-U8.1

date: illegal option -- 0 usage: date [-jnRu] [-d dst] [-r seconds|file] [-t west] [-v[+|-]val[ymwdHMS]] [-I[date | hours | minutes | seconds]] [-f fmt date | [[[[[cc]yy]mm]dd]HH]MM[.ss]] [+format] date: illegal option -- 0 usage: date [-jnRu] [-d dst] [-r seconds|file] [-t west] [-v[+|-]val[ymwdHMS]] [-I[date | hours | minutes | seconds]] [-f fmt date | [[[[[cc]yy]mm]dd]HH]MM[.ss]] [+format]

date in zpool_report.sh after upgrade to TrueNAS CORE

After upgrading to TrueNAS CORE, the zpool_report.sh script does return:

date: illegal option -- 0
usage: date [-jnRu] [-d dst] [-r seconds|file] [-t west] [-v[+|-]val[ymwdHMS]]
[-I[date | hours | minutes | seconds]]
[-f fmt date | [[[[[cc]yy]mm]dd]HH]MM[.ss]] [+format]
date: illegal option -- 2
usage: date [-jnRu] [-d dst] [-r seconds|file] [-t west] [-v[+|-]val[ymwdHMS]]
[-I[date | hours | minutes | seconds]]
[-f fmt date | [[[[[cc]yy]mm]dd]HH]MM[.ss]] [+format]

This will probably be connected to the OS upgrade and the scrubdate handling, but I do not understand the details.

Thank you very much for your great scripts!! The only issue I am missing at large - not only here - is handling of NVMe devices in smart reports.

email not working

smart_report.sh no longer sending email after upgrade to 11.3. I can send a test email from the GUI

get_hdd_temp.sh not reporting Device Model and Model Family on Toshiba drives

In my server, not FreeNAS bit running plain FreeBSD 13.0, I have a mix of Samsung, Micron and Toshiba SATA drives.
The Samsung and Micron drives are shown with their Device Model and Model Family in the output, but the Toshiba drives not at all. But when I run the command manually it shows the output just fine.

This is the script output for the drives:

=== DRIVES ===
da0: 31C [12.0TB] Z8B0A03KFP8G
da1: 32C [12.0TB] Z8A0A0N1FP8G
da2: 31C [12.0TB] Z8A0A0N8FP8G
da3: 32C [12.0TB] Z8B0A032FP8G
da4: 33C [12.0TB] Z8B0A03VFP8G
da5: 32C [12.0TB] Z8A0A0MBFP8G
da6: 38C [1.92TB] 18171D3C4771 Micron 5100 Pro / 52x0 (Micron_5200_MTFDDAK1T9TDN)
da7: 35C [1.92TB] 18171D3C4AD0 Micron 5100 Pro / 52x0 (Micron_5200_MTFDDAK1T9TDN)
ada0: 36C [250GB] S413NX0M707065K Samsung based SSDs (Samsung SSD 860 EVO M.2)
ada1: 35C [250GB] S413NX0M707143Y Samsung based SSDs (Samsung SSD 860 EVO M.2)

This is the line run manually. There is no Model Family present for this drive:

$ doas smartctl -i /dev/da0 | grep "Device Model"
Device Model: TOSHIBA HDWG21C

This is a full output of the -i command:

$ doas smartctl -i /dev/da0
smartctl 7.2 2020-12-30 r5155 [FreeBSD 13.0-RELEASE-p4 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model: TOSHIBA HDWG21C
Serial Number: Z8B0A03KFP8G
LU WWN Device Id: 5 000039 918c90e52
Firmware Version: 0601
User Capacity: 12,000,138,625,024 bytes [12.0 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: ACS-3 T13/2161-D revision 5
SATA Version is: SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Mon Oct 18 14:04:15 2021 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

I also have two NVME drives which are completely ignored in the output it seems.

If I can add anything to help fix this, please let me know.

Error on calculation of "Last Test Age" in smart_report.sh

The display of the Last Test Age was working for years without any issues.
On the last smart report i had this output:

+-------+------------------------+----+------+-----+-----+-------+-------+--------+------+----------+------+-----------+----+
|Device |Serial                  |Temp| Power|Start|Spin |ReAlloc|Current|Offline |Seek  |Total     |High  |    Command|Last|
|       |Number                  |    | On   |Stop |Retry|Sectors|Pending|Uncorrec|Errors|Seeks     |Fly   |    Timeout|Test|
|       |                        |    | Hours|Count|Count|       |Sectors|Sectors |      |          |Writes|    Count  |Age |
+-------+------------------------+----+------+-----+-----+-------+-------+--------+------+----------+------+-----------+----+
|ada0 ? |WD-************         |39  | 65620|  186|    0|      0|      0|       0|   N/A|       N/A|   N/A|        N/A|2732*|

...


########## SATA drive /dev/ada0 Serial: WD-************
########## Western Digital Red (WDC ************)

SMART overall-health self-assessment test result: PASSED

ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   182   173   021    Pre-fail  Always       -       3900
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       188
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   011   011   000    Old_age   Always       -       65620
 10 Spin_Retry_Count        0x0032   100   100   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   100   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       186
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       154
193 Load_Cycle_Count        0x0032   199   199   000    Old_age   Always       -       3283
194 Temperature_Celsius     0x0022   108   094   000    Old_age   Always       -       39
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0

No Errors Logged

Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
Short offline       Completed without error       00%        61         -

On further analysis i found out that the S.M.A.R.T. LifeTime(hours) counter seems to have reset itself

 /usr/local/sbin/smartctl -l selftest /dev/ada0
smartctl 7.2 2021-09-14 r5236 [FreeBSD 13.1-RELEASE-p2 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%        61         -
# 2  Extended offline    Completed without error       00%     65483         -
# 3  Short offline       Completed without error       00%     65334         -
# 4  Short offline       Completed without error       00%     65214         -
# 5  Extended offline    Completed without error       00%     65099         -
# 6  Short offline       Completed without error       00%     64974         -
# 7  Short offline       Completed without error       00%     64854         -
# 8  Extended offline    Completed without error       00%     64739         -
# 9  Short offline       Completed without error       00%     64590         -

In this resource i got the explanation that this counter is normally stored in a 16 bit field but could also differ for different HDD vendors: https://serverfault.com/questions/1041661/s-m-a-r-t-lifetime-hours-resetting-to-zero

For me i could fix the issue by adding a modulo function in the calculation
testAge=sprintf("%.0f", ((onHours % 65535) - lastTestHours) / 24);

FreeNAS-scripts/smart_report.sh

Line 131 in 06ccffb

testAge=sprintf("%.0f", (onHours - lastTestHours) / 24);

get-system-temps.pl "Argument "36.0C" isn't numeric in numeric le (<=) at get-system-temps.pl line 152"

Seems the perl script add C hard coded while the output from sysctl is already showing the C, resulting in the below error per CPU core:

=== CPU (16) ===
Argument "38.0C" isn't numeric in numeric le (<=) at get-system-temps.pl line 152.
CPU 0: 38.0CC
Argument "38.0C" isn't numeric in numeric le (<=) at get-system-temps.pl line 152.
CPU 1: 38.0CC
Argument "38.0C" isn't numeric in numeric le (<=) at get-system-temps.pl line 152.
CPU 2: 38.0CC
Argument "38.0C" isn't numeric in numeric le (<=) at get-system-temps.pl line 152.
CPU 3: 38.0CC
Argument "36.0C" isn't numeric in numeric le (<=) at get-system-temps.pl line 152.
CPU 4: 36.0CC
Argument "37.0C" isn't numeric in numeric le (<=) at get-system-temps.pl line 152.
CPU 5: 37.0CC
Argument "36.0C" isn't numeric in numeric le (<=) at get-system-temps.pl line 152.
CPU 6: 36.0CC
Argument "36.0C" isn't numeric in numeric le (<=) at get-system-temps.pl line 152.
CPU 7: 36.0CC
Argument "37.0C" isn't numeric in numeric le (<=) at get-system-temps.pl line 152.
CPU 8: 37.0CC
Argument "37.0C" isn't numeric in numeric le (<=) at get-system-temps.pl line 152.
CPU 9: 37.0CC
Argument "37.0C" isn't numeric in numeric le (<=) at get-system-temps.pl line 152.
CPU 10: 37.0CC
Argument "37.0C" isn't numeric in numeric le (<=) at get-system-temps.pl line 152.
CPU 11: 37.0CC
Argument "38.0C" isn't numeric in numeric le (<=) at get-system-temps.pl line 152.
CPU 12: 38.0CC
Argument "38.0C" isn't numeric in numeric le (<=) at get-system-temps.pl line 152.
CPU 13: 38.0CC
Argument "36.0C" isn't numeric in numeric le (<=) at get-system-temps.pl line 152.
CPU 14: 36.0CC
Argument "36.0C" isn't numeric in numeric le (<=) at get-system-temps.pl line 152.
CPU 15: 36.0CC