naplab/naplib-python

Tools and functions for neural data processing and analysis in python

naplib-python's Issues

Add GPU support for basic ML algorithms like TRF fitting

It would be great if cuML models were directly compatible with naplib's TRF function.

Probably need to either allow input data to be of type cupy.ndarray and check to see if this is the type, or just assume it is one case or the other.

Unify logging behavior through the ieeg pipeline

Expected Behavior

Currently, we log warnings, errors, info, and debug info using different methods in different parts of the code. Mainly using print functions and the logging library. We should unify all output so that it can be controlled from the same place. Ideally, the logging library.

Possible hard-coded befaft-time in process_ieeg

Expected Behavior

I think this line in process_ieeg
https://github.com/naplab/naplib-python/blob/main/naplib/naplab/process_ieeg.py#L260

and the next line
https://github.com/naplab/naplib-python/blob/main/naplib/naplab/process_ieeg.py#L261

essentially hard-coded a befaft period of 1 second on either side. The 1 should be swapped for the befaft[0] and befaft[1]

Add buffer for envelope extraction around start and end

Expected Behavior

The first few samples and the last few samples around phase/amplitude extraction get very distorted, so it would be great to add a buffer to befaft which is then removed after doing phase and amplitude extraction

Actual Behavior

Tell us what happens instead

Template Code

Paste the template code (ideally a minimal example) that causes the issue

Full Traceback

Paste the full traceback in case there is an exception

Your Environment

• Python version:
• naplib-python version:

error handling if don't pass labels to electrode_lags_fratio

Actual Behavior

If don't provide labels to electrode_lags_fratio(), then default=None but no error is thrown.

Template Code

lags = nl.segmentation.electrode_lags_fratio(data, field='resp')

Move up intermediate data downsampling in iEEG pipeline to load functions

Expected Behavior

Currently, downsampling to the intermediate sampling rate happens immediately after loading the raw data files. If we can move this resampling to within (some of) the load functions, it will improve performance substantially. However, this is only useful when loading data one electrode at a time, because resampling has to be done on the full signal and not chunks of it, to maintain same precision as we have now. Currently, this does not seem to be the case for any of the load functions.

process_ieeg computes spectrograms of more than just what are in stimorder

Expected Behavior

process_ieeg computes spectrograms of more than just what are in stimorder. It currently computes spectrogram for each audio file, and then just selects the ones that are in StimOrder. This is inefficient if you have a task which only uses some of the files.

Actual Behavior

Tell us what happens instead

Template Code

Paste the template code (ideally a minimal example) that causes the issue

Full Traceback

Paste the full traceback in case there is an exception

Your Environment

• Python version:
• naplib-python version:

Ability to read in BIDS data

Expected Behavior

Having the ability to read in data stored in BIDS format would be helpful for adoption by new users. BIDS is pretty well established across modalities.

https://bids.neuroimaging.io/

Append zeros to sounds and spectrograms

Expected Behavior

add zeros to start and end of stimulus in process_ieeg

import_data function won't accept out struct with only a single trial

Expected Behavior

Tell us what should happen

Actual Behavior

Fails if 'out' variable is a struct, rather than a struct array.

Template Code

Paste the template code (ideally a minimal example) that causes the issue

Full Traceback

Paste the full traceback in case there is an exception

Your Environment

• Python version:
• naplib-python version:

Normalize 'data' arg docstring is wrong

Actual Behavior

https://naplib-python.readthedocs.io/en/latest/_modules/naplib/preprocessing/preprocess.html#normalize

Parallelize TRF fitting over output channels

Expected Behavior

Since the TRF by default fits a separate model for each output channel, it would be easy to parallelize this loop.

Function for full preprocessing pipeline packaging

Single function/class for entire preprocessing pipeline given a config of parameters.

shaded and line plot colors not the same unless manually specified

Expected Behavior

In shadederrorplot(), the line and corresponding shaded region should be the same color without having to manually specify.

allow kwargs to be passed through in concat_apply

It would be nice if concat_apply took **kwargs or a kwargs_dict parameter which it passed through to the function being applied

function for re-referencing

Expected Behavior

General function which ideally should take in a graph (in matrix form), defining what electrodes to use when computing reference for that electrode. It should also take a method, such as 'avg', 'med', or 'pca'

For example, if 'resp' has 4 electrodes,

rereferenced_data = naplib.preprocessing.rereference(data, field='resp', arr=arr, method='avg')

where arr is a 4x4 matrix:

to define blocks for local rereferencing:

arr = [[1,1,0,0],
       [1,1,0,0],
       [0,0,1,1],
       [0,0,1,1]]

to define blocks for global rereferencing:

arr = [[1,1,1,1],
       [1,1,1,1],
       [1,1,1,1],
       [1,1,1,1]]

to define blocks for a weighted rereferencing:

arr = [[1,.5,.25,0],
       [.5,1,.5,.25],
       [.25,.5,.5,.5],
       [0,.25,.5,1]]

This would likely require another set of functions which help a user define these matrices based on simpler things.

For example

arr = naplib.utils.create_block_rereference_arr([1,1,2,2])
# arr = [[1,1,0,0],
#        [1,1,0,0],
#        [0,0,1,1],
#        [0,0,1,1]]

arr = naplib.utils.create_block_rereference_arr([1,1,1,1])
# arr = [[1,1,1,1],
#        [1,1,1,1],
#        [1,1,1,1],
#        [1,1,1,1]]

or to get it from a list/array of channel names:

arr = naplib.utils.create_rereference_arr_from_channelnames(['RTx1', 'RTx2', 'RTs1', 'RTs2'], method='amplifier')
# arr = [[1,1,0,0],
#        [1,1,0,0],
#        [0,0,1,1],
#        [0,0,1,1]]

visualization docs

https://naplib-python.readthedocs.io/en/latest/references/visualization.html#shaded-error-plot

Documentation for hierarchical cluster plot is missing description of "data" arg.

imSTRF description should say "colorbar is centered at zero", not data is centered at zero

Create Get-Started page on documentation

Would be great to have a page on the docs which details how the OutStruct works and how to get started with some simple stuff.

check_random_state not defined in naplib.model_selection.KFold

Expected Behavior

Kfold.split() when setting shuffle and random state should work.

Actual Behavior

KFold.split() raises NameError when calling split after setting random state.

Template Code

kfold = nl.model_selection.KFold(6, shuffle=True, random_state=1)
for t1, t2 in kfold.split([i for i in range(29)]):
    print((t1))

Full Traceback

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
/tmp/ipykernel_638517/928584509.py in <module>
      1 kfold = nl.model_selection.KFold(6, shuffle=True, random_state=1)
----> 2 for t1, t2 in kfold.split([i for i in range(29)]):
      3     print((t1))

~/naplab/GPT-Encoding/AnalysisCode/naplib/model_selection/model_selection.py in split(self, *args)
     82             )
     83 
---> 84         for train, test in super().split(data[0]):
     85             tmp = list(
     86                     chain.from_iterable(

~/anaconda3/envs/gpt_encoding/lib/python3.7/site-packages/sklearn/model_selection/_split.py in split(self, X, y, groups)
    338             )
    339 
--> 340         for train, test in super().split(X, y, groups):
    341             yield train, test
    342 

~/anaconda3/envs/gpt_encoding/lib/python3.7/site-packages/sklearn/model_selection/_split.py in split(self, X, y, groups)
     84         X, y, groups = indexable(X, y, groups)
     85         indices = np.arange(_num_samples(X))
---> 86         for test_index in self._iter_test_masks(X, y, groups):
     87             train_index = indices[np.logical_not(test_index)]
     88             test_index = indices[test_index]

~/anaconda3/envs/gpt_encoding/lib/python3.7/site-packages/sklearn/model_selection/_split.py in _iter_test_masks(self, X, y, groups)
     96         By default, delegates to _iter_test_indices(X, y, groups)
     97         """
---> 98         for test_index in self._iter_test_indices(X, y, groups):
     99             test_mask = np.zeros(_num_samples(X), dtype=bool)
    100             test_mask[test_index] = True

~/naplab/GPT-Encoding/AnalysisCode/naplib/model_selection/model_selection.py in _iter_test_indices(self, X, y, groups)
     46         indices = np.arange(n_samples)
     47         if self.shuffle:
---> 48             check_random_state(self.random_state).shuffle(indices)
     49 
     50         n_splits = self.n_splits

NameError: name 'check_random_state' is not defined

Your Environment

• Python version: 3.7
• naplib-python version:

logging throws error for python<3.9 in process_ieeg

Expected Behavior

logging throws error for python<3.9 in process_ieeg

dj-jayu/Img_link_to_local_markdown#2

Actual Behavior

Tell us what happens instead

Template Code

Paste the template code (ideally a minimal example) that causes the issue

Full Traceback

Paste the full traceback in case there is an exception

Your Environment

• Python version:
• naplib-python version:

Load data from edf files

Load raw data from edf files

Allow list of strings for process_ieeg's StimOrder

Expected Behavior

Instead of a file, allow a list of strings

Function to extract electrode localization information

Given a directory containing all the elec_recon files, minimally extract the following:

• fsaverage coordinates

If possible, also extract region labels, but there might not be a common format for this

add set_weights function to TRF

Expected Behavior

Would be great if you could manually set weights in TRF for easy simulation of STRFs

Load data from nwb format

Load raw data from NWB format

add stride parameter to sliding_window

Expected Behavior

Tell us what should happen

Actual Behavior

Tell us what happens instead

Template Code

Paste the template code (ideally a minimal example) that causes the issue

Full Traceback

Paste the full traceback in case there is an exception

Your Environment

• Python version:
• naplib-python version:

Missing docs for max_lags param

https://naplib-python.readthedocs.io/en/latest/references/segmentation.html#electrode-lags-from-f-ratio

Add nan_policy to ttest

Expected Behavior

Tell us what should happen

Actual Behavior

Tell us what happens instead

Template Code

Paste the template code (ideally a minimal example) that causes the issue

Full Traceback

Paste the full traceback in case there is an exception

Your Environment

• Python version:
• naplib-python version:

Enable changing the affinity metric in hierarchical cluster plot

naplib-python/naplib/visualization/plots.py

Line 404 in 1cf67c0

cluster = AgglomerativeClustering(n_clusters=n_clusters, affinity='euclidean', linkage='ward')

shaded error plot documentation error

Expected Behavior

Actual Behavior

Shaded error plot docstring has incorrect formatting resulting in Raises section being improperly displayed.

https://naplib-python.readthedocs.io/en/latest/references/visualization.html#shaded-error-plot

Function to do neural data preprocessing on raw data

Should minimally have the ability to perform the following:

• filter line noise (already in naplib.preprocessing)
• common referencing, including with mean, median, and pca (this might require a separate function added to naplib.preprocessing)

Add axis argument to hierarchical clustering

Expected Behavior

naplib.visualization.hierarchical_cluster_plot should have an axis argument which would be one of ['x', 'y', 'xy'] that specifies which axis of the data matrix to perform hierarchical clustering on. If x, the dendrogram will be on the top of the data figure (the current default). If y, the dendrogram would be to the left of the data figure. If xy, there would be two dendrograms, one above and one to the left of data.

Add legend to shadederrorplot

would be nice to automatically add legend strings to each line so if user calls plt.legend() it works

publish on conda to use GPU packages like cupy and cuML

publish on conda-forge

Add Merge/Join/Concat ability for Data object

Function to perform alignment for iEEG pipeline

Create a function which performs alignment based on a set of sounds/triggers (like a list of numpy arrays which are wav-data) and a single recording of the acoustics from the experiment. Might additionally require Stimulus Order information.

Add better bad-input checking after parsing outstruct args

Expected Behavior

After parsing args, there should be better and more descriptive errors thrown if data is not correct.

Actual Behavior

For example, if you try to pass a list of length 2 (when it should be an array of length 2), to responsive_ttest, the error comes up later, but it should really happen right after the inputs are parsed

https://github.com/naplab/naplib-python/blob/main/naplib/stats/responsive_elecs.py#L112

Template Code

Y = [np.random.rand(400,3) for _ in range(5)]
_, stats = responsive_ttest(resp=Y, befaft=[1,1], sfreq=100, alpha=0.01, random_state=1)

whereas no error happens if you correctly pass befaft as an array:

_, stats = responsive_ttest(resp=Y, befaft=np.array([1,1]), sfreq=100, alpha=0.01, random_state=1)

Your Environment

• Python version: 3.9
• naplib-python version: 1.4

update changelog for 0.1.1

Expected Behavior

Tell us what should happen

Actual Behavior

Tell us what happens instead

Template Code

Paste the template code (ideally a minimal example) that causes the issue

Full Traceback

Paste the full traceback in case there is an exception

Your Environment

• Python version:
• naplib-python version:

Add parallalized apply method to Data object

Expected Behavior

Would be nice to have a parallelized "apply" method for the naplib.Data class which would allow you to easily parallelize a function over trials of the Data object, since you typically have to loop through trials to apply a function to each.

Incorrect docstring for path in load_edf

Expected Behavior

Docs for load_edf function correspond to the load_tdt instead.

https://naplib-python.readthedocs.io/en/latest/references/io.html#load-edf

starting_fields not defined yet

naplib-python/naplib/out_struct.py

Line 256 in 29201c7

raise TypeError(f'Can only concatenate np.ndarrays, but found {type(starting_field[0])} in this field')

Allow specifying time range to `process_ieeg` to read a specific time span of the raw file

Expected Behavior

At the moment the load functions (load_edf, load_tdt, ...) support specifying a time range to read only a part of the data. But there is no way to leverage this when calling process_ieeg. This is most useful when the block of interest is a small part of the total recording file and we don't want to read the full file every time we run the pipeline. For example, when multiple experiments are contained in a single recording file.

Template Code

def process_ieeg(..., time_range: Union[int, Tuple[int, int]]=0, ...):
    ...

Tutorial notebooks on using mne visualizations with naplib outstructs

Expected Behavior

It would be great if there were tutorials on using mne's visualization tools along with naplib-python

Load data from TDT (tev/sev files)

load raw data from TDT machines, including ideally different versions of TDT

Add "average" parameter to stats.responsive_ttest

Would be great to have a parameter in naplab.stats.responsive_ttest to optionally control whether each segment (e.g. 1 second clip before and after onset) is averaged before being concatenated. Averaging may produce more consistent results, though not as good if you only have a few trials.

Change TRF fitting internal cross-validation to use naplib.model_selection.KFold

Internally, TRF performs cross-validation during fitting, but it uses a custom cross-validation splitter, so just convert this to naplib.model_selection.KFold to improve code re-use.

apply (without concat)

Expected Behavior

Add an apply method without concatenating for the array_ops module, because sometimes we don't want across-trial edge effects.

naplib.segmentation.electrode_lags_fratio doesn't properly document all parameters

Actual Behavior

The labels parameters within naplib.segmentation.electrode_lags_fratio does not have proper documentation.

Load raw stimuli from a directory of wav files

Load raw stimuli and return them, as well as meta-data like filenames

add ability to get manner of articulation dictionary

Actual Behavior

Would be nice to be able to get the manner_dict from this function:
https://github.com/naplab/naplib-python/blob/main/naplib/features/alignment_extras.py#L41