Reading in generic csv files

[clevengerkimberly]

Reading in generic csv files

[pasan-h]

Yes, unfortunately not yet possible. Meanwhile please use Claas' workaround.
We need to come up with a generic csv format perhaps including a header line(s) containing the device-ID. ActiPASS uses device-ID and then it can save and reuse device-calibration data.
See here to understand how ActiPASS parse ActiGraph files.
Anyone want to contribute with a generic CSV format definition and the corresponding ActiPASS parser? 😀

[claaslendt]

As far as I know, there is no possibility. Please correct me @pasan-h !

One way would be to reformat your acceleration data into an Actigraph .csv file format. I have done that in the past. You will need to specify the correct sampling frequency, start time and start date in the metadata header of the .csv file. I am happy to share some R code if you like.

[claaslendt]

@pasan-h I am more than happy to give it a go!

[claaslendt]

Here's my first draft for a generic CSV parser. The device ID and the sampling frequency are simply added as metadata in the first two lines of the CSV file. This can easily be done for a batch of files using R or Python. The sampling frequency could also be estimated from the timestamps otherwise. The acceleration axis would have to match the AX3 axis as specified by ActiPASS.

Even though the column names are not necessarily needed in the CSV file, I have added them for clarity.

I have also created a small test file: testdata.csv

function [Data, SF, deviceID] = readCSV(PATH)

% Read generic CSV files with specified format and ISO8601 timestamps
% 
%   Input:
%       PATH [string]: full file path as a string
%
%       Note:   Input CSV file must have the shape of (n_rows, 4). Where
%               n_rows is the number of sampled datapoints. The columns must be
%               named [time, x, y, z] and arranged in the exact order.
%       
%               The time column must be in ISO8601 format (e.g. '2024-04-20T14:30.010Z')
%
%               The acceleration axis must conform to the standard ActiPASS AX3
%               orientation (i.e. x-axis pointing down).
%
%               The first three lines in the CSV file must contain the
%               deviceID and the sampling frequency and the column names:
%               ID = <devideID>
%               SF = <sampling frequency>
%               time, x, y, z
%
%
%   Output:
%       Data          [Nx4]  datetime (Matlab datenum format) and 3D acceleration data
%       SF            [double] the sample frequency
%       deviceID      [string] the device ID

Data = [];
SF = NaN;
deviceID = NaN;

try
    % Open the file once and keep it open for reading both metadata and data
    fileID = fopen(PATH, 'r');
    
    % Read the first two lines for metadata
    idLine = fgetl(fileID);
    sfLine = fgetl(fileID);
    
    % Extract ID and SF values using correct indexing
    deviceIDParts = strsplit(idLine, '= ');
    deviceID = strtrim(deviceIDParts{2});
    SFParts = strsplit(sfLine, '= ');
    SF = str2double(strtrim(SFParts{2}));
        
    % Skip the header line
    fgetl(fileID);
    
    % Use textscan to read the data, reading datetime as text
    formatSpec = '%q%f%f%f';
    dataArray = textscan(fileID, formatSpec, 'Delimiter', ',', 'EmptyValue', NaN);
    fclose(fileID); % Ensure file is closed after reading
    
    % Convert datetime text to datenum
    timeText = dataArray{1};
    timeFormat = 'yyyy-mm-ddTHH:MM:SS.FFFZ';
    time = datenum(timeText, timeFormat);
    
    % Prepare the output data
    x = dataArray{2};
    y = dataArray{3};
    z = dataArray{4};
    Data = [time, x, y, z];
    
catch APE
    error("Error loading CSV file: " + APE.message);
end

[pasan-h]

Hi Claas,
Thanks very much for drafting the generic CSV import function. Now I have added the feature. I did some tests and realized the inclussion of a Time column (which is formatted as ISO8601 string) makes the CSV files about 80% big and also makes the loading considerably slower. Therefore I decided to ommit the time column. The consequence is that the format then only support resampled data to a fixed sample frequency. Since most accelerometer software supports resampling data to a fixed frequency before exporting to CSV anyway, I decided it's a good compromise. The source of the new function is here:
https://github.com/Ergo-Tools/ActiPASS/blob/main/utils/readGenericCSV.m
Cheers!

[pasan-h]

thanks Claas 🙏. I will try to add generic CSV import feature to to next ActiPASS release.
Things we need to think about:

• text parsing for big csv file would be slow (textscan). Perhaps we need to use a number format for time like CSVs from other devices. Perhaps the best is Unix epoch. Excel and matlab date numbers are also options. If we use Unix epochs, then we also need to define timezone of the measurement.
• The other options is if we can assume equi-distance samples (fixed sample-rate), then we do not need a time-column. But perhaps it's good to include a time column.
• I think some devices have missing data (like FirstBeat sensors with HR). Then perhaps we can add functionality to fill missing data to the parser. How about we use NaN to fill missing values?

[claaslendt]

Thanks, Pasan! I see it only as a first draft - especially since my MATLAB knowledge is fairly limited. From what I have read, textscan will be faster than e.g. readtable. But there will be a few other possibilities I am sure.

• I like the idea of switching the time to Unix time. That will be equally feasible. DateNum for example may be confusing for some users - since there is no straight forward way to convert timestamps to the DateNum format in R or Python as far as I know.
• Regarding the time: I'd argue to keep it in the file and use it for the subsequent resampling. Some users will assume their data has a fixed sample rate when infact it hasn't.
• The NaN option seems like a good idea. I guess we would have to make sure the missing data are all denoted the same in the CSV (i.e. empty value)