openneuroorg/openneuro

A free and open platform for analyzing and sharing neuroimaging data

OpenNeuro

About

OpenNeuro is a free and open platform for analyzing and sharing neuroimaging data. It is based around the Brain Imaging Data Structure specification.

Development setup

This project is managed with Lerna and Yarn. To get started, install Yarn and bootstrap the repo.

yarn install

You can run tests with yarn test at the top level of the project. For each package, yarn test --watch will interactively run the tests for changes since the last commit.

Before starting up the services, you will need to copy the example .env.example file to .env and config.env.example to config.env. Many of the values are optional, and most that aren't have default values included in their .example file. Required values below:

• JWT_SECRET in config.env must be set to a large random string.
• PERSISTENT_DIR in .env is an absolute path to a directory that will be used to store datasets. This should be a git-annex compatible filesystem and large enough to store some test datasets.

To setup Google as an authentication provider, register a new client app and set the following variables. For development use, you will create a new Google project with oauth credentials for a JavaScript client side app. "Authorized JavaScript Origins" is set to http://localhost:9876 and "Authorized Redirect URIs" is set to http://localhost:9876/crn/auth/google/callback for a site accessible at http://localhost:9876.

# Ending in .apps.googleusercontent.com
GOOGLE_CLIENT_ID=
# 24 character secret string
GOOGLE_CLIENT_SECRET=

podman-compose is used to run a local copy of all required services together.

# This will run podman-compose in the background (-d flag is --detach)
podman-compose up -d

For example, you can restart the server container with podman-compose restart server or view logs with podman-compose logs -f --tail=10 server.

Major Components

• OpenNeuro app - React frontend
• OpenNeuro server - Node.js GraphQL API
• OpenNeuro indexer - ElasticSearch indexer
• OpenNeuro components - ReactJS components library
• OpenNeuro CLI - Node.js command line tool
• OpenNeuro client - JavaScript client library used in CLI and App
• DataLad service - DataLad compatible dataset worker microservice
• bids-validator - BIDS validation library

JavaScript packages are published in the @openneuro npm namespace.

OpenNeuro Command-line utility tool

OpenNeuro supports a CLI tool based on nodejs for uploading and downloading OpenNeuro datasets.

openneuro's Issues

Cannot find lh.white.H

I'm experiencing issues running this new app (TRACULA) which seems related creation of symlinks:

ln -s rh.white.preaparc.H rh.white.H
Cannot find lh.white.H

For full log see https://openneuro.dev.sqm.io/datasets/ds000118/versions/00001?app=TRACULA&version=10&job=d6cbcaa1-acf3-4ae9-b3b4-05bab37605f8

Bulk of the computation done by this app is identical to what the freesurfer app is doing (which does not experience this error).

Support server side pagination in the client

Most endpoints return all data and are only paginated client side. This is part of the issue with #6 but it's a more general issue causing slow performance (especially if you are bandwidth constrained at all).

Analysis Status Issue for long running analysis with multiple jobs

The current behavior of the application is to update the overall status of the analysis once all the jobs in a multi-job analysis have been completed. This can cause problems if finished jobs sit in the SUCCEEDED or FAILED state on batch for longer than 24 hours waiting for other jobs to finish because the jobs are only persisted for 24 hours. Those jobs are no longer returned during status polling causing a potentially inaccurate status to be reported back to the client.
For example, this job below shows displays a status of SUCCEEDED even though the job status in mongo is RUNNING.

We should track the completion of individual jobs "real-time" so as to not lose track of the status of each individual job due to this issue.

Large uploads to S3 fail and are not automatically resumed

Sometimes a complete dataset is available in SciTran but it is not successfully copied to S3 when a job is started. This is more likely to occur as the dataset size goes up but can be worked around by canceling and retrying the job. It will usually succeed after this because the upload resumes where it left of and the underlying S3 or connectivity errors are infrequent.

To improve the upload performance and reliability we can move this SciTran to S3 upload to a queue and process the queue in the worker.

problem uploading large files

I am trying to upload a dataset that includes a large fMRI data file (~487 MB) and the upload has failed twice on that file. This is for https://openneuro.org/datasets/ds001040

Display docker image name and tag in results

Allow the creation of directories in datasets

Directories cannot be created or recreated if the last file is deleted.

Autocomplete on email address for sharing

it would be nice if the email entry on the sharing dialog would autocomplete with known email addresses

Analyses Sections Redesign

Currently the analyses sections in the dashboards are just list of analyses that link to the dataset that the analyses were run again. The dataset can be filtered by app and app version and sorted by date. See below.

There is probably a better way to display this data. Perhaps by grouping the analyses by dataset? Filtering by user who ran the analyses (for Public Dashboard)? Other ideas?

The expected outcome of this issue is to provide preliminary recommendations for UI updates that might improve the user experience with the Analyses sections of the application.

Event logs are slow to load

Load time: 90 sec.
Data transferred: 45 Mb (40 Mb from https://openneuro.org/crn/eventlogs?root=true)

This situation will only get worse. One solution would be to provide server side paging.

Add read-only public API

Backwards compatible versions of https://openfmri.org/dataset/api/

Listing all public datasets and their metadata.

This feature is used by nilearn and datalad. We need to make sure we don't break those tools.

Job ran out of space

The MRIQC execution of this dataset (https://openneuro.org/datasets/ds001060/versions/00001) ran out of space for a few subjects:

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/plugins/base.py", line 255, in run
  File "/usr/local/miniconda/lib/python3.6/site-packages/mriqc/bin/mriqc_run.py", line 325, in main
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/plugins/base.py", line 308, in _clean_queue
    result=result)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/plugins/multiproc.py", line 190, in _report_crash
    traceback=result['traceback'])
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/plugins/base.py", line 78, in report_crash
    crash2txt(crashfile, dict(node=node, traceback=traceback))
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/utils/filemanip.py", line 596, in crash2txt
    fp.write(''.join(record['traceback']))
OSError: [Errno 28] No space left on device

Writing to path: /usr/local/src/mriqc/work

Event Logs Updates

• need to add dataset upload event to event logs.
• need to display some more useful information in the event logs UI

Boolean app parameters do not translate to checkboxes

results in

instead of a checkbox.

If the checkbox is checked this should result in inclusion of --ica flag if it is not checked it should not be included.

Dataset ds000140 Failing to Upload

Re: the dataset here: https://openfmri.org/dataset/ds000140/

It consistently stalls while uploading to OpenNeuro and then fails after repeated attempts to upload (please see attached screenshots).

Polling never updates status when the pending / running time difference is larger than 24 hours

This mostly happens when jobs are waiting in the queue between analysis levels. A participant level job can complete and 24 hours passes before the group level starts. This will get stuck and the status from batch will never be updated.

Job succeeds, but results tab not presents

https://openneuro.dev.sqm.io/datasets/ds000118/versions/00001?app=TRACULA&version=7&job=4de25fb8-0d60-4323-bfd5-59e66b3afb05

Previously I noticed that for some (other) jobs the results were loaded asynchronously and the tab would show up after some delay. This time it does not show up at all.

Store job logs outside of CloudWatch

CloudWatch was very convenient but to support multiple backends, we'll need a way to collect those logs and display them from our own datastore. For Batch, it should be populated from CloudWatch but we should no longer require CloudWatch outside the lifetime of a running job.

PermissionError: [Errno 13] Permission denied: '/output/data/derivatives'

https://openneuro.dev.sqm.io/datasets/ds000105/versions/00001?app=mindboggle&version=1&job=418a5959-43ec-459f-a614-2c41ad701f4e

^@Namespace(analysis_level='participant', bids_dir='/snapshot/data', output_dir='/output/data', participant_label=['13'])
['13']
running
subject_label is 13
/snapshot/data/13/anat/sub-13_T1w.nii.gz
/snapshot/data/13/anat/sub-13_ses-*_T1w.nii.gz
images are ['/snapshot/data/sub-13/anat/sub-13_T1w.nii.gz']
Create missing output directory /output/data/derivatives/mindboggle
Traceback (most recent call last):
  File "/opt/conda/bin/mindboggle123", line 122, in <module>
    os.makedirs(OUT)
  File "/opt/conda/lib/python3.5/os.py", line 231, in makedirs
    makedirs(head, mode, exist_ok)
  File "/opt/conda/lib/python3.5/os.py", line 241, in makedirs
    mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/output/data/derivatives'
Traceback (most recent call last):
  File "~/code/run.py", line 76, in <module>
    run_mindboggle(t1, label, args.output_dir)
  File "~/code/run.py", line 25, in run_mindboggle
    check_call(cmd)
  File "/opt/conda/lib/python3.5/subprocess.py", line 581, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['mindboggle123', '/snapshot/data/sub-13/anat/sub-13_T1w.nii.gz', '--id', 'sub-13', '--out', '/output/data/derivatives/mindboggle', '--working', '/output/data/scratch']' returned non-zero exit status 1

This could be a problem with the app itself, but I don't see how... Creating this directory should be possible.

Upload Failure - Recurring Issue

I am currently trying to upload this dataset: https://openneuro.org/datasets/ds001067

It has failed three times thus far, but continues to make incremental progress each time I resume the upload. This is a recurring issue with each dataset I have previously uploaded.

My laptop is connected to a power source, has no disruption in connection, and has not gone to sleep.

Please advise. Thanks.

Handle unavailable endpoints without crashing

The primary issue is /api/projects from #6. We should catch any failures like this and warn the user to better handle low bandwidth connections.

Google indexing

Currently content of OpenNeuro is not indexed by Google properly

Use BIDS app versions instead of OpenNeuro app versions

https://gitlab.sqm.io/stanford_crn/openneuro/issues/190

Temporary workaround #61

Display "Acknowledgements" field in the successful job results

We need to show users how to acknowledge the authors of the app. Best place to do it in the job result box.

This is relatively high priority if we want to make app developers happy.

Deposit snapshots of public datasets in OpenNeuro

All public datasets and their snapshots should be deposited in the s3://openneuro bucket with the same file hierarchy as we use right now.

Enable Firefox support

Firefox 49+ supports the needed multifile upload APIs. To enable it, we need to do a once over to fix some minor CSS issues and update the browser check.

• Firefox CSS fixes.
• Remove check preventing uploads from Firefox

Provide a better way to get large results

Providing the S3 URL for the results and instructions for using that URL to download your results will make getting results out of the system easier.

A dependency of solving this is we should use signed URLs to prevent excessive reuse of the S3 URLs (redownloading the same dataset many times) and protect private datasets.

Develop HTCondor submit service

To interact with the queue from #50 on HTCondor systems, we need to run the consumer on a submit server local to the cluster. The scope of this should be limited to obtaining a job from the queue, translating it to DagMan / Submit data and submitting that.

I think this should use the Python bindings but one of the other interfaces may prove better if we can share some code with the existing Node implementation.

Jobs should automatically be stopped after 48 hours

Jobs running for more than 48 hours should be failed automatically to prevent stuck or very resource intense jobs from running indefinitely.

Display "Support" field in the failed job results

When job fails we want to instruct users how to get in touch with the app developers.

Uploading hangs on the last file during upload

From the user:

The upload process is stuck, so the dataset is not complete.

I deleted my draft and tried again, and sadly the same problem occurred. It seems to be stuck on the last file, although it has been uploading over a day.
Has anyone else encountered this problem before?

I've tried, but was not able to replicate the issue on my machine with the same dataset (which I will share with @nellh and @JohnKael).

App Definition Parameters UI Improvements

The interface for adding parameters to an app definition is currently a little confusing and needs improvement. The current UI is displayed below.

A common issue with this UI is the adding of a parameter but forgetting to press ADD button before submitting the app definition. This interface should be redesigned to make the user experience better and less confusing.

As part of this work, we should also address hidden options, as this will add another checkbox to the interface. See below for details from Chris.

"Some BIDS App command line arguments (for example those regarding cpu and memory restrictions) should be always the same for all jobs and users submitting jobs should not be able to change them. The easiest way to implement this is to add a checkbox next to each parameter to determine if this parameter will be visible to users submitting jobs."

Webpack CSS is not minified

Small issue with the new webpack config. The production build should minify the CSS.

Reduce /api/projects bandwidth requirements and latency

When loading https://openneuro.org/dashboard/datasets being logged in as [email protected]

app.min.1b7726f0.js:sourcemap:54 Uncaught TypeError: Cannot read property 'body' of undefined
    at app.min.1b7726f0.js:sourcemap:54
    at a (app.min.1b7726f0.js:sourcemap:54)
    at app.min.1b7726f0.js:sourcemap:54
    at d.callback (app.min.1b7726f0.js:sourcemap:45)
    at d.crossDomainError (app.min.1b7726f0.js:sourcemap:45)
    at XMLHttpRequest.n.onreadystatechange (app.min.1b7726f0.js:sourcemap:45)

This seems like a critical bug.

Job fails, logs unaccessible "The specified log stream does not exist."

https://openneuro.org/datasets/ds001038/versions/00001?app=BARACUS&version=3&job=da564531-76cd-402f-809b-37994f1c3223

"download all logs" fails with "Failed - Server problem"

Originally reported by @poldrack

No UI feedback for missing required fields

When trying to submit a job without providing a required field (BARACUS, license key) clicking the "Start" button leads to no action, but warning or other information is presented to the user (such as highlighting missing required inputs).

Upgrade to React 15

Some front end bugs (especially some UI performance issues) depend on upstream fixes in React or one of the other libraries requiring newer React to upgrade. The major things that need to be done here are:

• Resolve dependency issues (switch to yarn and fix the dependency list to accurately reflect installed deps).
• Write some tests for the UI components to help validate changes for the update.
• Refactor routing to upgrade to react-router v4.
• Handle all the now invalid uses of the React API and missing packages that were refactored out of React.
• Extra padding below navbar in some views.
• Switching snapshots does not correctly trigger the loading state and animations.
• Active states for many tabs are not working.

Setup better performance tests

I'd like to setup a second dev server to do performance regression testing, which will require some automatic deployment work. Lighthouse CI looks excellent for this as Lighthouse is already one of the tools being used to optimize performance and headless Chrome provides fairly realistic performance values for actual usage of the OpenNeuro site.

• Deploy to dev and perf servers on every commit. Dev should track master, perf should track whatever branch is being tested.
• Setup Lighthouse with CircleCI.
• Graph that data somewhere so we can see how it's changing over time (comparing releases for example).

Collect execution metrics for each analysis

From @poldrack:

it would be useful to know what kind of instances are used for freesurfer, MRIQC, and fMRIPREP

That's something that can be saved but isn't at the moment. The metrics will need to be per-task since a given analysis can be scheduled over several host systems. Some ideas for what can be collected:

• Instance used (type, EC2 ARN)
• Real run time (CPU time, not wall clock)
• Percent utilization (CPU/memory allocated vs used)

@poldrack Anything else you'd like to see for this?

intermittent button failure on analysis dropdown

I have had an intermittent problem where the start button on an analysis fails to do anything when clicked. It seems to occur only when I run an analysis after just having run another within the same session on that window. refreshing the window seems to resolve the problem. low priority but I can imagine that it would flummox some people.

Fix inability to delete directories

I want to remove one entire subject from a dataset. I am able to delete the files within the subject folder, but unable to remove the subject directory, which causes a validation failure. should include ability to delete folders.

S3 configuration is only updated by new app versions

This forces updates to app versions when any changes are made to the S3 buckets, rather than applying that configuration to a submitted job. The configuration needs to be added to job submission and removed from app definition registration.

Editing input arguments - hidden checkbox glitch

On dev:

Clicking the "Hidden for Users" does not seem to have an effect.

Uncaught TypeError: Cannot read property 'length' of undefined

When trying to load https://openneuro.org/datasets/ds001049

Uncaught TypeError: Cannot read property 'length' of undefined
    at t.value (/app.min.4f999727.js:50)
    at t.value (/app.min.4f999727.js:50)
    at t.value (/app.min.4f999727.js:50)
    at c._renderValidatedComponentWithoutOwnerOrContext (/app.min.4f999727.js:38)
    at c._renderValidatedComponent (/app.min.4f999727.js:38)
    at c.mountComponent (/app.min.4f999727.js:37)
    at Object.mountComponent (/app.min.4f999727.js:40)
    at v.mountChildren (/app.min.4f999727.js:39)
    at v._createContentMarkup (/app.min.4f999727.js:38)
    at v.mountComponent (/app.min.4f999727.js:38)

We have a user stuck on this - it would be great to get it fixed in a timely manner.

Parameter Key field not cleared after adding param

When adding a parameter in the admin app definition form, the key field is not cleared after adding parameter resulting in a prepopulated parameter form.

Transfer remaining data from OpenfMRI

This would allow us to finalize the process of porting all data from openfmri to openneuro and maintain consistency of accession numbers.

Add links to jobs in the event log

For debugging and monitoring purposes it would be useful to be able to go from an event log entry to the dataset/job page.

Create an internal jobs queue

Jobs are started out of process but the queue mechanism is very specific to Batch endpoints. It should be replaced with a generic queue describing pending jobs that can be translated into requests to Batch or HTCondor.

Linux / Chrome issue with submit jobs dropdowns

Noticed this in #48 and it's Linux specific, so making it a new issue to fix.

Distinguish between failed and successful completed jobs in the event log

There are two use cases for this feature:

• as @poldrack suggested we would like to see if the percentage of successful jobs changes over time
• when we see a failing job in the logs we might want to investigate it even if the user did not report it via the support page