datalad-datasets/hcp-functional-connectivity

Preprocessed functional connectivity MR data from the WU-Minn HCP1200 dataset. **More info in README.**

hcp-functional-connectivity's Issues

Incomplete data for sub-208226

Dear datalad team,

It appears that subject 208226 is missing the rfMRI_REST2_RL_hp2000_clean.nii.gz file under /208226/MNINonLinear/Results/rfMRI_REST1_RL/ and in the documentation it is registered that this participant has all for rs-fmri acquisitions. Would it be possible to include it?

Thank you very much in advance!

Best regards,
Lya

Incomplete data for 3 subjects

Specifically 150928, 186444, 188347. This is a bug in the superdataset: datalad-datasets/human-connectome-project-openaccess#9

"s3 key refused" error on `datalad get`

Hi! I've successfully cloned the dataset following the instructions in the readme, but when I go to "get" a participant's data, I get an error that the s3 key was refused.

get(error): 171330/MNINonLinear/Results/rfMRI_REST1_LR/rfMRI_REST1_LR_Atlas_stats.txt (file) [Failed to download from any of 1 locations ['S3 refused to provide the key for HCP_1200/171330/MNINonLinear/Results/rfMRI_REST1_LR/rfMRI_REST1_LR_Atlas_stats.txt from url s3://hcp-openaccess/HCP_1200/171330/MNINonLinear/Results/rfMRI_REST1_LR/rfMRI_REST1_LR_Atlas_stats.txt?versionId=xKe91c2Yr0GgJ6laGI0y9VEPTX1jIcfl -caused by- S3ResponseError: 403 Forbidden\n']

If I try to just download it using the aws cli, and the s3 link from that output, I also get an error.

(fc) mphagen@nid005744:/pscratch/sd/m/mphagen/hcp-functional-connectivity> aws s3 cp s3://hcp-openaccess/HCP_1200/171330/MNINonLinear/Results/rfMRI_REST1_LR/rfMRI_REST1_LR_Atlas_stats.txt?versionId=xKe91c2Yr0GgJ6laGI0y9VEPTX1jIcfl . 

fatal error: An error occurred (404) when calling the HeadObject operation: Key "HCP_1200/171330/MNINonLinear/Results/rfMRI_REST1_LR/rfMRI_REST1_LR_Atlas_stats.txt?versionId=xKe91c2Yr0GgJ6laGI0y9VEPTX1jIcfl" does not exist 

But, if I trim off the trailing alphanumeric junk, then I AM able to download it. So it doesn't seem to be an issue with my s3 keys being set incorrectly.

aws s3 cp s3://hcp-openaccess/HCP_1200/171330/MNINonLinear/Results/rfMRI_REST1_LR/rfMRI_REST1_LR_Atlas_stats.txt . 

download: s3://hcp-openaccess/HCP_1200/171330/MNINonLinear/Results/rfMRI_REST1_LR/rfMRI_REST1_LR_Atlas_stats.txt to ./rfMRI_REST1_LR_Atlas_stats.txt

My version of datalad is 0.19.2.

I was also able to replicate this in the full HCP datalad dataset, so it's not something that's specific to this dataset.

edit:

output of datalad wtf

[WARNING] Could not determine filesystem types due to KeyError(None) 
# WTF
## configuration <SENSITIVE, report disabled by configuration>
## credentials 
  - keyring: 
    - active_backends: 
      - PlaintextKeyring with no encyption v.1.0 at /global/homes/m/mphagen/.local/share/python_keyring/keyring_pass.cfg
    - config_file: /global/homes/m/mphagen/.config/python_keyring/keyringrc.cfg
    - data_root: /global/homes/m/mphagen/.local/share/python_keyring
## datalad 
  - version: 0.19.2
## dependencies 
  - annexremote: 1.6.0
  - boto: 2.49.0
  - cmd:7z: 16.02
  - cmd:annex: 10.20230408-g5b1e8ba77
  - cmd:bundled-git: 2.40.1
  - cmd:git: 2.40.1
  - cmd:ssh: 8.4p1
  - cmd:system-git: 2.40.1
  - cmd:system-ssh: 8.4p1
  - humanize: 4.7.0
  - iso8601: 2.0.0
  - keyring: 24.2.0
  - keyrings.alt: 4.2.0
  - msgpack: 1.0.3
  - platformdirs: 3.10.0
  - requests: 2.31.0
## environment 
  - GIT_PYTHON_REFRESH: quiet
  - LANG: en_US.UTF-8
  - PATH: /global/common/software/nersc/current/jupyter/ex/23-06/nersc-utils:/global/homes/m/mphagen/miniconda3/envs/fc/bin:/global/homes/m/mphagen/miniconda3/condabin:/global/common/software/nersc/bin:/global/common/software/nersc/pm-2021q4/easybuild/software/Nsight-Systems/2022.2.1:/global/common/software/nersc/pm-2021q4/easybuild/software/Nsight-Systems/2022.2.1/bin:/global/common/software/nersc/pm-2021q4/easybuild/software/Nsight-Compute/2022.1.1:/opt/nvidia/hpc_sdk/Linux_x86_64/22.7/cuda/11.7/compute-sanitizer:/opt/nvidia/hpc_sdk/Linux_x86_64/22.7/cuda/11.7/bin:/opt/nvidia/hpc_sdk/Linux_x86_64/22.7/cuda/11.7/libnvvp:/opt/nvidia/hpc_sdk/Linux_x86_64/22.7/profilers/Nsight_Compute:/opt/nvidia/hpc_sdk/Linux_x86_64/22.7/profilers/Nsight_Systems/bin:/opt/cray/pe/perftools/23.03.0/bin:/opt/cray/pe/papi/7.0.0.1/bin:/opt/cray/pe/gcc/11.2.0/bin:/opt/cray/pe/craype/2.7.20/bin:/opt/cray/pe/mpich/8.1.25/ofi/gnu/9.1/bin:/opt/cray/pe/mpich/8.1.25/bin:/opt/cray/libfabric/1.15.2.0/bin:/usr/local/bin:/usr/bin:/bin:/usr/lib/mit/bin:/opt/cray/pe/bin
  - PYTHONFAULTHANDLER: 
## extensions 
## git-annex 
  - build flags: 
    - Assistant
    - Webapp
    - Pairing
    - Inotify
    - DBus
    - DesktopNotify
    - TorrentParser
    - MagicMime
    - Benchmark
    - Feeds
    - Testsuite
    - S3
    - WebDAV
  - dependency versions: 
    - aws-0.22.1
    - bloomfilter-2.0.1.0
    - cryptonite-0.29
    - DAV-1.3.4
    - feed-1.3.2.1
    - ghc-9.0.2
    - http-client-0.7.13.1
    - persistent-sqlite-2.13.1.0
    - torrent-10000.1.1
    - uuid-1.3.15
    - yesod-1.6.2.1
  - key/value backends: 
    - SHA256E
    - SHA256
    - SHA512E
    - SHA512
    - SHA224E
    - SHA224
    - SHA384E
    - SHA384
    - SHA3_256E
    - SHA3_256
    - SHA3_512E
    - SHA3_512
    - SHA3_224E
    - SHA3_224
    - SHA3_384E
    - SHA3_384
    - SKEIN256E
    - SKEIN256
    - SKEIN512E
    - SKEIN512
    - BLAKE2B256E
    - BLAKE2B256
    - BLAKE2B512E
    - BLAKE2B512
    - BLAKE2B160E
    - BLAKE2B160
    - BLAKE2B224E
    - BLAKE2B224
    - BLAKE2B384E
    - BLAKE2B384
    - BLAKE2BP512E
    - BLAKE2BP512
    - BLAKE2S256E
    - BLAKE2S256
    - BLAKE2S160E
    - BLAKE2S160
    - BLAKE2S224E
    - BLAKE2S224
    - BLAKE2SP256E
    - BLAKE2SP256
    - BLAKE2SP224E
    - BLAKE2SP224
    - SHA1E
    - SHA1
    - MD5E
    - MD5
    - WORM
    - URL
    - X*
  - operating system: linux x86_64
  - remote types: 
    - git
    - gcrypt
    - p2p
    - S3
    - bup
    - directory
    - rsync
    - web
    - bittorrent
    - webdav
    - adb
    - tahoe
    - glacier
    - ddar
    - git-lfs
    - httpalso
    - borg
    - hook
    - external
  - supported repository versions: 
    - 8
    - 9
    - 10
  - upgrade supported from repository versions: 
    - 0
    - 1
    - 2
    - 3
    - 4
    - 5
    - 6
    - 7
    - 8
    - 9
    - 10
  - version: 10.20230408-g5b1e8ba77
## location 
  - path: /global/u1/m/mphagen
  - type: directory
## metadata.extractors 
## metadata.filters 
## metadata.indexers 
## python 
  - implementation: CPython
  - version: 3.9.17
## system 
  - distribution: sles/15.4/n/a
  - encoding: 
    - default: utf-8
    - filesystem: utf-8
    - locale.prefered: UTF-8
  - filesystem: 
    - CWD: 
      - path: /global/u1/m/mphagen
    - HOME: 
      - path: /global/homes/m/mphagen
    - TMP: 
      - path: /tmp
  - max_path_length: 276
  - name: Linux
  - release: 5.14.21-150400.24.46_12.0.73-cray_shasta_c
  - type: posix
  - version: #1 SMP Tue Jun 13 16:43:10 UTC 2023 (9c4698c)