Comma added to end of csv output file when 'Terminal width too small, trimming output' warning seen.

[davecoutts]

Comma added to end of csv output file when 'Terminal width too small, trimming output' warning seen.

[davecoutts]

For anybody striking this issue, here is a snippet that permits the ingestion of dstat csv log data into pandas/jupyter without errors brought on by the variable column width issue.

The column alignment in this code snippet is specific to the the output produced by dstat run with the -Tcdilmnprsy --proc-count options.

import pandas as pd

dstat_csv_file = '/tmp/dstat_Tcdilmnprsy_proc_dstat_master.csv'

column_names = ['timestamp',
                'cpu_usr', 'cpu_sys', 'cpu_idl', 'cpu_wai', 'cpu_hiq', 'cpu_siq',
                'dsk_read', 'dsk_writ',
                'intp_a', 'intp_b', 'intp_c', 
                'la_1m', 'la_5m', 'la_15m', 
                'mem_used', 'mem_buff', 'mem_cach', 'mem_free', 
                'net_recv', 'net_send', 
                'proc_run', 'proc_blk', 'proc_new', 
                'io_read', 'io_writ', 
                'swap_used', 'swap_free', 
                'sys_int', 'sys_csw', 
                'proc_total']

df = pd.read_csv(dstat_csv_file, skiprows=6, usecols=range(0,31)) # usecols used to slice a consistent data set and avoid csv variable column width issue.
df.columns = column_names  # Setting column names here rather than with the 'read_csv' 'names' option as read_csv errors when variable column widths seen.
                           # See https://github.com/dagwieers/dstat/issues/154
    
df['timestamp'] = pd.to_datetime(df['timestamp'],unit='s')       # epoch to datetime format
df['timestamp'] = df['timestamp'].values.astype('datetime64[s]') # Remove millisecond precision
df.set_index('timestamp', inplace=True)

df.head(2)