🐛 Missing default yml in v1.8.5 Docker Image

[tjlagrow]

🐛 Missing default yml in v1.8.5 Docker Image

[amygutierrez]

Hello,
Thank you for bringing this to our attention. Maybe you can try adding the flag --preconfig default and see if that helps. So your command would look like

sudo cpac run /home/tjlagrow/Documents/ADNI/sNC_f/bids_dir /home/tjlagrow/Documents/ADNI/sNC_f/outputs test_config --platform docker --preconfig default

Please let me know if that helps or if any other error arises.

[amygutierrez]

Thank you for your detailed comments! I was able to run your command on my system without any hanging issues or redundancy issues (Ubuntu 18.04.1 LTS).
Are you still using the latest version of CPAC (v1.8.5)? To upgrade the version, you can do cpac --platform docker --tag latest pull and you can also add --tag latest in your cpac run command to ensure you are using the latest version.
Also, can you show maybe a tree of how your bids directory looks like? I can try mimicking it to see if I run into the same issue.

[tjlagrow]

Thank you for the prompt response, I appreciated it!
I was able to use the latest docker image when using the --preconfig default flag. However, I am still running into the same issue of virtually no processing. This was not an issue previously and I am unsure what to do to help correct this. I did several tests both on my system and docker for both ram health and harddrive health - both are healthy. I also attempted a fresh install of Ubuntu 20.04 LTS to no added benefit. Is there maybe something else I'm missing?

Previously, I was able to use the test_config command and the process would finish in under a minute. With the new update, the process is hanging up for minutes on each nipype.workflow process. I have tried a great number of times and I have still yet to complete one subject using test_config. I will post one of my run's output. I am trying to use the test_config command on 3 HCP participants, each with 4 scans. I expect 12 outputs but after 15 minutes, I have yet to get through 1 scan of "connecting...".
cpac_docker_processing_issue.txt I will continue to let this current process go to see what happens and will report here if/when the process finishes.

Additionally, when I process a bids directory, the overarching data_config and file pipeline_config are generated for each subject AND scan, generating quite a lot of redundancy.

Please let me know if you need any further information.

[tjlagrow]

I was able to get through 1 scan with the output log (pypeline.log). What took usually under a minute is now taking over 3 hours. I am a little lost on what is going on here or if I need to update a specific package in the docker image (nipype?) or if something else locally is going on? I stopped the process since this is just a test_config run. I took another scan I've run before and I'm trying to run the participant command. I will report back if there is an issue with the processing when the initial nipype.workflow connecting *... finishes.

I am now running:
sudo cpac -vv run /home/tjlagrow/Documents/hcp_data/bids_dir_small /home/tjlagrow/Documents/hcp_data/outputs participant --platform docker --preconfig default \ --data_config_file /home/tjlagrow/Documents/hcp_data/outputs/cpac_data_config_99fd7508_2023-06-08T15-31-42Z.yml \ --pipeline_config_file /home/tjlagrow/Documents/hcp_data/outputs/cpac_pipeline_config_99fd7508_2023-06-08T15-31-42Z.yml \ --mem_gb 60.0 --n_cpus 16

[tjlagrow]

Thank you for your quick comments!
I am unsure what changed but I was able to run the processes on two systems on different linux distros (Ubuntu 20.04 LTS and CentOS) without a processing issues. This was true for both CPAC v1.8.4 and v1.8.5. I am not sure why it fixed the issue, but using sudo python3 -m cpac ... ended up doing the trick. I am not sure if it was a pathing issue for specific packages but that shouldn't be the case with the dockerized code.

In terms of a file structure, attached is a printout. This is the result of ls -R which is a little messy.
file_structure.txt

[shnizzedy]

@tjlagrow I'm glad you were able to get cpac to run. The issue here I believe is just a little asynchrony in the release cycles of the various C-PAC packages ― C-PAC itself v1.8.5 is available, but the latest version's documentation isn't locked yet, and the changes necessary for the cpac convenience wrapper to work seamlessly with v1.8.5 are not yet merged / released / published.

In case anyone finds this issue when C-PAC is ahead of cpac and the docs, the nightly documentation for running directly in Docker or in Singularity is another way to run the latest (or nightly) image when cpac isn't yet supporting that version. Or if you aren't necessarily wanting to use the bleeding edge version, you can specify a less-recent version --tag like

cpac --tag release-v1.8.4 run /home/tjlagrow/Documents/ADNI/sNC_f/bids_dir /home/tjlagrow/Documents/ADNI/sNC_f/outputs test_config

to run the second-most-recent version of C-PAC (the most recent version currently supported by cpac)

Or, if you do want bleeding-edge and like to live dangerously, you can pip install cpac from an unreleased branch or commit like

pip install git+https://github.com/FCP-INDI/cpac.git@cpac_resources

or

pip install git+https://github.com/FCP-INDI/cpac.git@cabac8b820075876cfc28966071306b9314c139e

to install a development version of the wrapper.

Duplicate of FCP-INDI/cpac#35