[Feature Request] IOstat Disk Service Time support

[nmonvisualizer]

[Feature Request] IOstat Disk Service Time support

[hpresnall]

I can consider adding this. Do you have sample output and the command line flags you ran with to generate it?

[luizlinux]

I sure do!

iostat -DTl [ interval] [ count ]

If used without the interval and count arguments [like vmstat], the information displayed by the iostat command will be a single report containing statistics for the time since system startup (boot).
However, this would not be interesting, since the ideal thing would be to have a collection going on on the machines that could be later interpreted and graphed by nmon visualizer.

The fields which would be interesting to graph from IOstat are:

• Read avg serv - Average service time per read transfer.
• Read min serv - Minimum service time per read transfer.
• Read max serv - Maximum service time per read transfer.
• Write avg serv - Average service time per write transfer.
• Write min serv - Minimum service time per write transfer.
• Write max serv - Maximum service time per read transfer.

I have done this with some degree of success using something like the following script, which produces a CSV file of a period of 24h divided into 5 seconds intervals.

LANG=C;echo "Time,Disk,RAvgSrv,RMinSrvR,RMaxSrv,WAvgSrv,WMinSrv,WMaxSrv";iostat -DTl 5 17280 | grep ^hdisk | awk '{print $25,$1,$8,$9,$10,$14,$15,$16}' | sed 's/\ /,/g' > /nmon/diskServTime.csv

I then download the CSV, import it into LibreOffice or Excell, then use pivot tables to meshup the data and then I can finally graph using line graphs, but man, how cumbersome it is!

One word of advice though: If using Dell EMC's sorry excuse of a multipath driver, PowerPath, where the disks are grouped into hdiskpower names, iostat will not show statistics for the hdiskpower disks. In that case you have to look up and average the statistics of each hdisk disk which composes each hdiskpower disk in order to see its response time health.
To sum up: Doing this with PowerPath is a mess, not impossible, but a huge pain.
But lets not get into that now! I think that for a first atempt of implementing this we should stick to the regular hdisk disk format used by most hdisk drivers.

[hpresnall]

What version of iostat are you running (iostat -V)?
I don't see -D, -T or -l as options even on the latest from https://github.com/sysstat/sysstat

Also, looking at the man page (https://github.com/sysstat/sysstat/blob/master/man/iostat.in#L293) the svctm field is marked as untrustworthty. Also see https://www.xaprb.com/blog/2010/09/06/beware-of-svctm-in-linuxs-iostat/. Wondering if these other times suffer from the same issues?

[luizlinux]

I thought you picked up from the hdisk named disks, but I was talking about AIX iostat. ;)
https://www.ibm.com/support/knowledgecenter/en/ssw_aix_72/com.ibm.aix.cmds3/iostat.htm

[hpresnall]

iostat is a real mess to parse and AIX has so many conflicting options. Based on this and other issues I am seeing in Linux, I think I need to redo a lot of this parser. May take me a while, but I guess that's ok since I took over a year to respond to this request ;)

As part of that, I am not going to worry about AIX 5.3 or 6.1 since both are now out of support. I'm also going to limit the supported sets of switches. Here's what I am thinking:

1. -T - "basic"; also allows -a for adapter info or -f for file system info (mutually exclusive)
2. `-Tam - "basic with path & adapter"
3. -TDl - "extended"

All the above would support and encourage the use of -s so the hostname is output along with the cpu.

I only have access to a small system with no multipath. So, if you have a system with more than one path available (maybe with EMC), I'd like the see some sample output from -Tams. I would probably be able to sum the paths for all disks in that case.

[luizlinux]

iostat is a real mess to parse and AIX has so many conflicting options. Based on this and other issues I am seeing in Linux, I think I need to redo a lot of this parser. May take me a while, but I guess that's ok since I took over a year to respond to this request ;)

Agreed, IOstat is a mess to parse on any platform. As to the time it takes to implement this, given the quality of the current code, I think you have all it takes and more. ;)

As part of that, I am not going to worry about AIX 5.3 or 6.1 since both are now out of support. I'm also going to limit the supported sets of switches.
...

Agreed, although the proposed supported switches are present in back level versions of AIX as well, this means that your code might be backwards compatible (which is nice). I happen to have [totally against my will] all sort of old AIXs in my work place, from 5.2 onwards I've got them all. I could test them to let you know about the backwards compatibility, just for curiosity.

I only have access to a small system with no multipath. So, if you have a system with more than one path available (maybe with EMC), I'd like the see some sample output from -Tams. I would probably be able to sum the paths for all disks in that case.

How about I send you two files?

• One from a server with regular MPIO multipath disks [which encapsulates all paths under one hdisk name].
• One from a server with Powerpath, EMC's crapy multipath driver [which encapsulates all paths under one hdiskpower name but leaves all paths visible as hdisk names (awful idea)].

In order to provide you with not so large files, I'm thinking about collecting for one hour using 5 seconds intervals. What do you think?

[hpresnall]

At 5 second intervals, 10 or 15 minutes is more than enough. Thanks!

[luizlinux]

Here you go my man!:
Server_with_MPIO_iostat_-Tams_5_120.out.gz
Server_with_MPIO_iostat_-TDls_5_120.out.gz
Server_with_Powerpath_iostat_-Tams_5_120.out.gz
Server_with_Powerpath_iostat_-TDls_5_120.out.gz

The file names should give away what their content is.
All taken from AIX 7.1 LPARs. Gziped them otherwise they would be huge.

You will notice that -TDls on powerpath systems brings no info on powerpath disks and -Tams on powerpath systems doesn't show the hdiskpower disks. Have I told you how crapy this driver is? lol

Anyway, enjoy and let me know if I can be of any further assistance!

[hpresnall]

I have this working on the files you sent. Would you be able to pull and build the latest commit to try it on other files?

Looking at the Powerpath data, I am not sure you can aggregate some of that data, which is probably why they didn't do any of it. Total reads and writes, yes. But what's the average service time for an aggegate disk? Is it the average of all the averages, a weighted sum, what? Not sure what the statistically valid value would be.

If you really need that data, it may be better to script something that aggregates it how you want. Once the files are parsed in NMONVisualizer, you can have it output CSV data by right clicking on either the server name and doing Save to CSV or right clicking on each disk folder and selecting Copy to put it on the clipboard. You can also use the command line with --rawdata.

[luizlinux]

Hello.
Just built from git, you really outdone yourself!

Could we have a grouping graph, like DISKREADSERV and DISKWRITESERV from nmon parsing, where I can see many disks at once?
I mean, it would be nice to see many disks at once instead of one by one, specially on servers with lots of disks. I think that could be split into several graphs, also like how its done with DISKREADSERV and DISKWRITESERV from nmon where they receive the suffix 1, 2, 3 etc.

Cheers

[luizlinux]

P.S.
By many disks I mean, something like 40 disks per graph.
Currently DISKREADSERV and DISKWRITESERV are split into graphs with too many disks and graphing fails, unless you have a movie theater for screen. hehehehe

[hpresnall]

I've actually never run NMON with that many disks. I didn't realize it did that.

Can you see if a custom report will work for you (View -> Custom Report then File -> Load)? Take a look at iostat_report.xml.

I'd strongly encourage you to change the regex in <type regex="IOStat (Device|Disks|Paths/Disks) \((.*)\)" aliasByGroup="2" /> to something that limits the disks by changing the .* part to something like hdiskpower.*. Trying to graph all the disks seems to completely hang the program and even just doing hdisk1.* takes multiple seconds per graph to display.

It's also possible to create a chart per disk with something like this report. Just make sure you click the Chart Multiplexing By Type option in the custom report window.

Note you need to pull and rebuild the latest for this to work. I changed how some of the iostat fields were being named to more closely match the man page names.