nipype/pydra-tutorial

Pydra tutorial

pydra-tutorial's Issues

Dataset for nilearn tutorial

Currently we are directly converting the nilearn examples directly using nilearn.datasets
It would be good to expand it to a full analysis.
We will need to find a preprocessed dataset on task-fMRI for this.

container build error on binder

We should fix the container, raw log is here

convert this repo to a jupyterbook

@djarecka and @yibeichan - it may be worthwhile to spend a bit of time to convert this repo to a jupyterbook. it shouldn't be too hard and it will make the notebooks easier to diff and read since they will be in myst/markdown, and the site will be generated through continuous integration ensuring that the notebooks work with the latest versions of software.

Workflow for two-level GLM (nilearn tutorial)

Hello, I'm working on doing a two-level GLM nilearn tutorial for Pydra. But I've been having problems with a node in the workflow for more than two weeks (@djarecka, @htwangtw, and I have discussed it over past weeks). We've tried a couple ways to debug but haven't solved the problem yet. Here is a summary about what's happening:

In this two-level GLM tutorial, the analytical logic is:

1. download data (a task function)
2. run first level for each subject (a workflow)
3. starting the second level estimation using results from the 2nd step (a task function)
4. multiple statistical testings.... (other tasks...)
5. step 3&4 together can be set up as a workflow, but let's talk about task function first for the sake of the current issue.

I write the whole workflow using nilearn and make sure the code itself is error-free. Each pydra task function can run successful as a standalone task. When I connect tasks into workflow, errors come up.

1. The workflow works fine to get first level estimation
2. The problem happens at secondlevel_estimation see here cell 18.
   2.1 The input of secondlevel_estimation is a list of firstlevel model, the outputs are (a) secondlevel mask from SecondLevelModel() and (b) secondlevel stats estimations
   2.2 If run secondlevel_estimation as a task (outside of workflow), it prints results fine.
   2.3 If run secondlevel_estimation as a node in the workflow, it runs and can print out outcomes at every step, but it CANNOT return results. Here (last line) is the error 'NoneType' object has no attribute 'errored'
   2.4 Since secondlevel_estimation can't return results, it has problems linking to the next node, so we will get error as graph is not empty, but not able to get more tasks - something is wrong (e.g. with the filesystem) see the last cell output here

Now the question is why this secondlevel_estimation node can't return results even if the output at each of its step can be print out.

updates for the tutorials

• cont_dim, see nipype/pydra#532

IsADirectoryError" only occurs when using pydra

Hi all,

I am currently trying to build some workflows with pydra. They should include functions of a library, which we have developed. one step of the workflow would be reading in the config file. Consequently, I tried to set up a really simple workflow for doing that. However, when executing the workflow I get error the following error message:

"name": "IsADirectoryError",
"message": "[Errno 21] Is a directory: '/data2/egapontseva/MEG_QC_stuff/data/from_openneuro/ds003483'",

The workflow looks as follows:

import nest_asyncio

nest_asyncio.apply()

import pydra
import meg_qc  
from meg_qc.meg_qc_pipeline import make_derivative_meg_qc
from meg_qc.source.initial_meg_qc import get_all_config_params, Epoch_meg

# Define the task
@pydra.mark.task
def get_config_params(configfile):
    config_dict = get_all_config_params(configfile)
    return config_dict



# Define the Pydra workflow
workflow_name = "initial_processing"
input_spec = ["configfile"] 
initial_processing_wf = pydra.Workflow(name=workflow_name, input_spec=input_spec)

initial_processing_wf.add(get_config_params(name='read_config', configfile=path_to_config))

# Specify the output of the workflow
initial_processing_wf.set_output([
    ("config_dict", initial_processing_wf.read_config.lzout.out)
])

# Submit the workflow for execution using the 'cf' plugin
with pydra.Submitter(plugin='cf') as sub:
    sub(initial_processing_wf)

# View the output of the workflow
print(initial_processing_wf.result())

The problem is that when just running the function isolated like this: config_dict = get_all_config_params(path_to_config) it works perfectly and does not produce any error.

I would appreciate your help, since error does not really make sense to me. The directory_path in the dictionary is actually intended to be a path to a directory and that is what produces the error, in the case of running it with pydra, to my understanding.

I am using pydra 0.22.

Typo in FunctionTask tutorial: "pf" instead of "of" under section 2.2. Setting the input

fixing the exercise format

I believed I used the exercise2 extension for the hiding the solutions, but didn't work right now

7_twolevel_glm_nilearn doesn't work well on jupyterbook

It seems that we have some problems of running this tutorial on jupyerbook. I suspect that it's related to datalad again. We probably need to provide fake username and email for it. I'll fix it soon.

Error in the illustrative image for Workflow section 4.5 (Setting a splitter for nodes)

Under section 4.5 of Workflow, the workflow input is x=[3,5,7] which is provided as Lazy Inputs to both the mean and power tasks as x_list and a respectively, which implies both x_list and a equals [3,5,7] , evident from the code:

wf8 = pydra.Workflow(name='wf8', input_spec=['x'], x=[3, 5, 7])
wf8.add(mean(name='mean', x_list=wf8.lzin.x))
# adding a task that has its own splitter
wf8.add(power(name='power', a=wf8.lzin.x).split('a'))

But the illustrative image for the same code shows both x_list and a with value [3,5,8]:

update tutorial to reflect environment class updates

@djarecka - is this still valid? https://nipype.github.io/pydra-tutorial/notebooks/5_intro_shelltask.html#dockertask

Binder environment not working for the tutorial notebooks.

The Binder environment for the tutorial notebooks has stopped working.
Expected response: The Binder environment for the Jupyter notebooks
Actual response: Jupytext Markdown format

Figures are missing, or have wrong outputs

../figures/nested_workflow.png

Both images have wrong outputs associated with inputs provided

removing specific version from the requirements

We should remember that we will have to remove pydra==0.23.0a0 at some point

It appears the `nested_workflow.png` image in `1_intro_pydra.ipynb` is still missing. Couldn't find it in `figures/`..

It appears the nested_workflow.png image in 1_intro_pydra.ipynb is still missing. Couldn't find it in figures/..

Originally posted by @SamGuay in #3 (comment)

Typo in Workflow section 4.2 (Workflow as a node)

In the code block provided for section 4.2 of Workflow there is a minor typo error in the comment:

# connecting wfa to the input from the main workflow
wf2a.inputs.x = wf5.lzin.x

wf2a is incorrectly written as wfa