conp-pcno / conp-portal Goto Github PK
View Code? Open in Web Editor NEW:bar_chart: The CONP data portal
Home Page: https://portal.conp.ca/
License: MIT License
:bar_chart: The CONP data portal
Home Page: https://portal.conp.ca/
License: MIT License
Need to write the roadmap and beta release specs. Plan to open between July and August
Need to include a higher res picture (logo) for the navbar brand section
As discussed with @paiva and @joeyzhou98, #51 will result in a minimal, usable interface for pipeline search. That minimal version should be extended to comply to the mockup initially designed. @joeyzhou98 could add more information following our discussion.
On the Forums section of the CONP portal, it would be nice to explore:
Assess and select suitable 3rd party authentication service(s) based on initial requirements, e.g. ORCID.
We should streamline the processing of CONP datasets with CONP pipelines, possibly by reviving https://github.com/CONP-PCNO/conp-pipeline
Assess and select suitable framework based on initial requirements, e.g. Drupal, Joomla, etc.
Ensure that when a user try to upload a new dataset, a message remind the user about dataset type (open, restricted, registered, ...) requirement connect with CONP ethics and governance.
to respect ethics and privacy of people
Here is a block of code to add to the Prevent-AD description
<h2>Documentation</h2>
<ul>
<li>
<a href="https://conp.ca/wp-content/uploads/2019/04/PREVENT-AD-short-description.pdf">Prevent-AD short description</a>
</li>
<li>
<a href="https://conp.ca/wp-content/uploads/2019/04/MRI-protocol-for-open-science.pdf">MRI protocol for Open Science</a>
</li>
</ul>
<h2>List of Publications</h2>
<ol>
<li>
<p>Brain properties predict proximity to symptom onset in sporadic Alzheimer's disease. Vogel JW, Vachon-Presseau E, Pichet Binette A, Tam A, Orban P, La Joie R, Savard M, Picard C, Poirier J, Bellec P, Breitner JCS, Villeneuve S; Alzheimer’s Disease Neuroimaging Initiative and the PREVENT-AD Research Group. Brain. 2018 Apr 23.</p>
<a href="https://www.ncbi.nlm.nih.gov/pubmed/29688388">doi: 10.1093/brain/awy093.</a>
</li>
<li>
<p>Bi-directional Association of Cerebrospinal Fluid Immune Markers with Stage of Alzheimer’s Disease Pathogenesis. Meyer PF, Savard M, Poirier J, Labonté A, Rosa-Neto P, Weitz TM, Town T, Breitner, J; Alzheimer’s Disease Neuroimaging Initiative; PREVENT-AD Research Group. J Alzheimers Dis. 2018 Apr 11.</p>
<a href="https://www.ncbi.nlm.nih.gov/pubmed/29660934">doi: 10.3233/JAD-170887.</a>
</li>
<li>
<p>Subjective cognitive decline is associated with altered Default Mode Network connectivity in individuals with a family history of Alzheimer’s Disease, Verfaillie SCJ, Pichet Binette A, Vachon-Presseau E, Tabrizi S, Savard M, Bellec P, Ossenkoppele R, Scheltens P, van der Flier WM, Breitner JC, Villeneuve S for the PREVENT-AD Research Group. Biological Psychiatry: Cognitive Neurosciences and Neuroimaging. May 2018.</p>
<a href="https://www.ncbi.nlm.nih.gov/pubmed/29735156">doi: 10.1016/j.bpsc.2017</a>
</li>
<li>
<p>Proximity to Parental Symptoms Onset and Amyloid-β Burden in sporadic Alzheimer disease. Villeneuve S, Vogel JW, Gonneaud J, Pichet Binette A, Rosa-Neto P, Gauthier S, Bateman RJ, Fagan AM, Morris JC, Benzinger TL, Johnson SC, Breitner JC, Poirier J, for the Prevent-AD group. JAMA Neurology. 2018 May 1.</p>
<a href="https://www.ncbi.nlm.nih.gov/pubmed/29482212">doi: 10.1001/jamaneurol.2017.5135.</a>
</li>
<li>
<p>Regionally specific changes in the hippocampal circuitry accompany progression of cerebrospinal fluid biomarkers in preclinical Alzheimer’s disease, Tardif CL, Devenyi G, Amaral RSC, Pelleieux S, Poirier J, Rosa-Neto P, Breitner J, Chakravarty MM; PREVENT-AD Research Group., Hum Brain Mapp. 2017 Nov 21.</p>
<a href="https://www.ncbi.nlm.nih.gov/pubmed/29164798">doi: 10.1002/hbm.23897</a>
</li>
<li>
<p>Deficit in Central Auditory Processing as a Biomarker of Pre-Clinical Alzheimer’s Disease, Tuwaig, Miranda, Savard, Mélissa, Jutras, Benoît, Judes Poirier, Louis Collins, Pedro Rosa-Neto, David Fontaine and John C.S. Breitner for the PREVENT-AD Research Group, Journal of Alzheimer's Disease, vol. 60, no. 4, pp. 1589-1600, 21 Aug 2017.</p>
<a href="https://www.ncbi.nlm.nih.gov/pubmed/28984583">doi: 10.3233/JAD-170545</a>
</li>
<li>
<p>Validation of a Regression Technique for Segmentation of White Matter Hyperintensities in Alzheimer's Disease. Dadar M, Pascoal TA, Manitsirikul S, Misquitta K, Fonov VS, Tartaglia MC, Breitner J, Rosa-Neto P, Carmichael OT, Decarli C, Collins DL. IEEE Trans Med Imaging. 2017 Aug;36(8):1758-1768.</p>
<a href="https://www.ncbi.nlm.nih.gov/pubmed/28422655">doi: 10.1109/TMI.2017.2693978</a>
</li>
<li>
<p>Odor identification as a biomarker of preclinical AD in older adults at risk. Lafaille-Magnan ME, Poirier J, Etienne P, Tremblay-Mercier J, Frenette J, Rosa-Neto P, Breitner JCS; PREVENT-AD Research Group. Neurology. 2017 Jul 25;89(4):327-335.</p>
<a href="https://www.ncbi.nlm.nih.gov/pubmed/28659431">doi: 10.1212/WNL.0000000000004159</a>
</li>
<li>
<p>Alzheimer’s Progression Score’: Development of a Biomarker Summary Outcome for AD Prevention Trials J.-M. Leoutsakos, A.L. Gross , R.N. Jones, M.S. Albert, J.C.S. Breitner (2016) The Journal of Prevention of Alzheimer’s Disease - JPAD© Volume 3, Number 4, 2016.</p>
<a href="https://www.ncbi.nlm.nih.gov/pubmed/29034223">doi: 10.14283/jpad.2016</a>
</li>
<li>
<p>Rationale and Structure for a new Center for Studies on Prevention of Alzheimer’s Disease (StoP-AD) J.C.S. Breitner, J. Poirier, P.E. Etienne1, J.M. Leoutsakos for the PREVENT-AD Research Group (2016) The Journal of Prevention of Alzheimer’s Disease - JPAD© Volume 3, Number 4, 2016.</p>
<a href="https://www.ncbi.nlm.nih.gov/pubmed/29199324">doi: 10.14283/jpad.2016.121.</a>
</li>
<li>
<p>Test-retest resting-state fMRI in healthy elderly persons with a family history of Alzheimer's disease. Orban P, Madjar C, Savard M, Dansereau C, Tam A, Das S, Evans AC, Rosa-Neto P, Breitner JC, Bellec P; PREVENT-AD Research Group. Sci Data. 2015 Oct 13;2:150043.</p>
<a href="https://www.ncbi.nlm.nih.gov/pubmed/26504522">doi: 10.1038/sdata.2015.43.</a>
</li>
<li>
<p>INTREPAD: a randomized trial of naproxen to slow progress of pre-symptomatic Alzheimer’s disease, Pierre François Meyer*, MSc; Jennifer Tremblay-Mercier*, MSc; Jeannie Leoutsakos, PhD; Cécile Madjar, MSc; Marie-Élyse Lafaille-Maignan, PhD; Melissa Savard, MSc; Pedro Rosa-Neto, MD, PhD; Judes Poirier, PhD**; Pierre Etienne, MD**; John Breitner, MD, MPH** for the PREVENT-AD research group. Neurology, 2019 Apr 5 [Epub ahead of print].</p>
<a href="https://www.ncbi.nlm.nih.gov/pubmed/30952794">doi: 10.1212/WNL.0000000000007232.</a>
</li>
</ol>
It would be ideal to deploy some analytics tool and some event tracking systems to better understand user behavior and, if possible, find out what it is being done with the data downloaded.
Event tracking tags should be consistent with those to be used by the LORIS platform so that we are consistent across both platforms.
Some event tracking tools include:
Some dashboard tools include:
Integrated pipelines could be exported automatically and periodically to ReproNim: https://github.com/repronim/containers A cron job could pull the Boutiques descriptors, extract their container image, and push it to that datalad repository.
We want for users to be able to download datasets that they have authorization to download to be able to do so without additional logins. Primarily, this is a currant issue for LORIS datasets, but also will need to be addressed for datasets in other platforms (e.g. BrainCode).
We want for users to not necessarily need to go to other platforms to get the data unless we truly need to.
For LORIS, quick and dirty, we decided to create a global user that would be used to access data through the portal.
Transition from portal.conp.io to portal.conp.ca
It breaks for NeuroLibre icon, download buttons, and data search aspects. Need to show cards instead
In order to facilitate development and deployment, create the capability to distinctly set up a development vs. production environment.
This way developers can set up a local env without the need to have use a dedicated VM. Should ensure that the dev environment is deployable quickly on a laptop.
using a well defined configuration definition for flask app, potentially multiple ones for each environment, and allowing a .flaskenv file to define local variables specific to the machine with flask-dotenv.
Currently, the datasets in our portal are statically updated at install time through a kludgy migration. This should be dynamically populated through calls to whatever backend API that we are using, or pull from a registry process.
The portal should be dynamic in how it presents its datasets.
We could support both, but this is an open issue that would be good to discuss amongst the team and TSC before we move forward.
There is a need to reorganize and clean up our current set of CONP VMs. Currently each have been deployed with a single use account that is being shared amongst several people, and each of them doesn't at this point serve a clear purpose.
In order for the project to move from a demo platform to a full fledged production environment with a geographically shared development effort, we need to bring these into the MCIN fold and reevaluate how our developers interact with the infrastructure.
As we are moving into taking the CONP development from a demonstration effort to a long-term sustained production environment, we need to reevaluate how we securely and appropriately utilize MCIN infrastructure. Ideally, the VMs created should be for testing and hosting deployments rather than as shared environments for development (as this is not a scalable solution, especially for remote development). Additionally, it is incredibly insecure to have shared VMs with only one user account that are not tied to centralized user accounting system, as auditing activity is impossible.
Design initial portal UX/UI, including
The file all_descriptors.json
is not downloaded in the .cache
repository locally, therefore when I search for pipelines, it gives an error that it cannot find that file
The file all_descriptors.json
should be downloaded first before launching the application to avoid having the error problem.
Traceback (most recent call last):
File "/home/s/Work/MNI/CONP/dev/conp-portal/venv/lib/python3.5/site-packages/flask/app.py", line 2309, in __call__
return self.wsgi_app(environ, start_response)
File "/home/s/Work/MNI/CONP/dev/conp-portal/venv/lib/python3.5/site-packages/flask/app.py", line 2295, in wsgi_app
response = self.handle_exception(e)
File "/home/s/Work/MNI/CONP/dev/conp-portal/venv/lib/python3.5/site-packages/flask/app.py", line 1741, in handle_exception
reraise(exc_type, exc_value, tb)
File "/home/s/Work/MNI/CONP/dev/conp-portal/venv/lib/python3.5/site-packages/flask/_compat.py", line 35, in reraise
raise value
File "/home/s/Work/MNI/CONP/dev/conp-portal/venv/lib/python3.5/site-packages/flask/app.py", line 2292, in wsgi_app
response = self.full_dispatch_request()
File "/home/s/Work/MNI/CONP/dev/conp-portal/venv/lib/python3.5/site-packages/flask/app.py", line 1815, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/home/s/Work/MNI/CONP/dev/conp-portal/venv/lib/python3.5/site-packages/flask/app.py", line 1718, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/home/s/Work/MNI/CONP/dev/conp-portal/venv/lib/python3.5/site-packages/flask/_compat.py", line 35, in reraise
raise value
File "/home/s/Work/MNI/CONP/dev/conp-portal/venv/lib/python3.5/site-packages/flask/app.py", line 1813, in full_dispatch_request
rv = self.dispatch_request()
File "/home/s/Work/MNI/CONP/dev/conp-portal/venv/lib/python3.5/site-packages/flask/app.py", line 1799, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/home/s/Work/MNI/CONP/dev/conp-portal/app/routes.py", line 417, in pipeline_search
with open(all_desc_path, "r") as f:
FileNotFoundError: [Errno 2] No such file or directory: '/home/s/.cache/boutiques/all_descriptors.json'
This happened in Ubuntu-based systems. I was able to replicate the bug 3 times. It does not seem to affect Mac users.
It would behoove us to adopt a coding standard to ensure that our code is well written and extensible. I am open to suggestions, but PEP8 is pretty standard for Python based projects. This would require us to run pep8 on code before we accept pull request.
https://www.python.org/dev/peps/pep-0008/
https://pypi.org/project/pep8/
Any opinions?
Coding standards are good!
<!-- styles the nav container -->
.navbar.navbar-inverse-red {
position: fixed;
z-index: 99999;
top: 0;
left: 0;
width: 100%;
min-height: 60px;
background: #fff;
border-bottom: 2px solid;
border-bottom-color: #FF0000;
box-shadow: 2px 2px 8px rgba(0,0,0,.5);
}
<!-- overwrites bootstrap margin on nav container -->
.navbar-nav {
margin: 0 0 20px;
}
<!-- overwrites styles for list item anchors -->
.navbar-nav>li>a {
position: relative;
display: block;
padding: 10px 10px;
font-weight: 400;
text-decoration: none;
line-height: 1.6em;
}
<!-- restyles hamburger. the website uses font-awesome so this makes it look similar -->
.navbar-toggle .icon-bar {
display: block;
width: 30px;
height: 5px;
border-radius: 8px;
background-color: #343434;
}
<!-- link to google fonts for the head of the site -->
<link href="https://fonts.googleapis.com/css?family=PT+Sans+Narrow:400,700|PT+Sans:400,700" rel="stylesheet">
<!-- set the fonts of the site -->
body {
color: #343434;
font-family: 'PT Sans', sans-serif;
font-size: 16px;
font-size: 1rem;
line-height: 1.5;
}
h1, h2, h3, h4, h5, h6 {
clear: both;
font-family: 'PT Sans Narrow', sans-serif;
line-height: 1.3em;
margin-top: 1rem;
margin-bottom: 1rem;
font-weight: 400;
}
h1 { font-size: 2.4em; }
h2 { font-size: 2.2em; }
h3 { font-size: 2em; }
h4 { font-size: 1.8em; }
h5 { font-size: 1.6em; }
h6 { font-size: 1.4em; }
<!-- note - you'll need to reset the navigation items to the default lower case except first letter -->
Need to add link to URL image. Where is it going to point to?
Establish iteration user stories, focused on:
The project should be tested on Travis, using pytest and coveralls.
Some tasks will require to have a CONP-related email account (notifications and user support)
The synced view of Datalad datasets in the portal is awesome! But I think it would benefit from the following improvements:
JB tried to log in with being registered first and broke the system. Need to put some checking in place.
The conp-portal VM needs a production capable web server set up and running. Currently this does not seem to be configured correctly.
I will:
I am getting :
https://portal.conp.ca/undefined?page=NaN&max_per_page=undefined
when I click on the arrow to go to the next page
The login pages have a different base and need to be introduced into the styleguide.
So that we have a definitive styleguide for the whole website.
Pipeline objects have an online-platform-urls
list property that refers to Web platforms where the pipeline is installed. When this property is defined, pipeline search should add an icon linking to a platform in online-platform-urls
in the pipeline record.
online-platform-urls
has more than 1 entry, the portal should select an entry containing the string "cbrain" if available. Otherwise, it should select the first entry in the list. If multiple URLs exist that contain the string "cbrain", it should choose the first one.Pipeline zenodo.3267250
currently has an online-platform-urls
property defined with an element containing "cbrain".
It is the first step toward enabling pipeline execution on CBRAIN, see #63. As part of #58, @shots47s added a CBRAIN URL to zenodo.3267250
but it is not showing in the portal
The pipeline JS record reuses the dataset record. The link should be added to the CBRAIN logo there:
If the URL doesn't point to CBRAIN, a generic logo should be used, such as https://fontawesome.com/icons/globe?style=regular
@joeyzhou98 would you like to take on this when done with your current task?
Do we require a login in the platform to access all of the data, even completely open datasets, or do we require a login or not (independent of what that login is).
Right now, the portal does not require a login to see and access the data that is unrestricted and open. It only requires it to see registered data (in the current case, just the PreventAD dataset). We should decide once and for all, what the model is.
Have users login to use the portal and access all of the data
a. Advantage: Can track all downloads to an individual user.
Have users only need to login to get access to data that is not unrestricted an open
a. Advantage: More open
b. Disadvantage: We will only be able to log access to datasets via IP address where the requests are coming from.
Right now, there is no tie of the Stats to the Datasets, and they are not dynamically updated at all in the portal.
The stats should be updatable and validated to actions in the portal.
We should tie the stats to the Dataset via a one-to-one relationship and add hooks in the portal to updating when actions occur.
The updating of pipelines data from Zenodo happens at add places in the portal workflow by spawning threads. These leads to inconsistent datasets, and the file based nature of this seems a bit fragile.
Should we have all of the data for the portal translated to the database, so that we have models to manipulate or is it best to rely on the cache files that are occurring now? The database may be easier to index with something like elastic search.
rewrite the system so that updating the pipeline data modifies the records in the database rather than being created on the fly from files in a .boutiques directory.
Overview of the search engine:
Search through tools by using the bosh command line (form Boutiques). Boutiques should act as an "app store" framework where it retrieves tools through bosh search. The unique tool identifier (DOI) is returned from Zenodo and can be used to allow the user to download the tool.
One key element about the search is that it should use web caching, as individual bosh searches are quite slow.
To help in usability of the portal, and remind user that a dataset need registered access instead of letting them think it is just not available for some (unknown) reason.
users that are not logged-in can still see registered datasets but the icon to download is grayed out and unclickable.
Create a place to build command line management tools for the portal such as database operations such as backup, restore, and seeding as well as other functions we may need in the future
having a succinct place to put these tools can be easier to develop in the future and facilitate testing and translation.
create a separate cli.py that can be imported into the app and give the proper application context so that testing can also be done on a mock database without rewriting any code or configurations.
Check out what I did in the https://github.com/shots47s/NRC-Map-Converter-Flask for guidance.
Currently the bootstrap css is allowing images to change aspect ration as they are resized. I suggest adding the following to the css
img {
width: 100%;
height: auto;
}
If user is not authenticated or the dataset is private, there should not be an tag on the download button, instead just invoke an alert or redirect to the login page
We should prepare a basic demo on how to process data with a pipeline in a Neurolibre notebook. We could for instance do brain extraction of a few subjects using zenodo.1482743
. The way to do it would be to use Boutiques' desc2func feature, in a way similar to:
> from boutiques.descriptor2func import function
> mcflirt = function('zenodo.2602109')
> mcflirt(in_file='/home/glatard/data/test.nii.gz')
This may have already been started or looked at, but creating a definitive style-guide / CSS for the portal so that there is not a bunch of custom CSS floating around.
Having a definitive style guide will ameliorate development and ensure that as we move forward, the portal has a uniform look and feel.
Review of the current CSS's in the repo to clean up and ensure we are moving in the right direction.
Tools found through the portal will be identified as available in CBRAIN when their online-platform-urls
property (a list) contains a URL with the string "cbrain" in it. Many pipelines returned by search aren't currently available in CBRAIN, and if they are, it's likely that their descriptor doesn't include the URL of a CBRAIN portal. Pipelines returned by bosh search
should be reviewed, added to CBRAIN plugins whenever relevant, and republished.
To ensure that people are aware of restriction to use registered data.
To help enforce ethics.
Need to add the multi-layer registration steps: including agreeing to the ToS, community agreement/expectations, code of conduct, etc.
Ensure that datasets submitted to CONP through Zenodo have equivalent access restriction.
Access restriction displayed and selected on Zenodo match what is advertized on CONP.
Zenodo has different restriction on access.
#51 will result in the user being redirected to CBRAIN to run pipelines. We should discuss whether we would like to have a tighter integration with CBRAIN. Steps could be:
I think (1) is doable and (2) is not really necessary but that would require some discussion.
Consider adding Altmetric scores next to datasets listed on the Conp website. Altmetric scores are an attention score used for tracking publications. Adding an Altmetric score next to the dataset will help the owner know where their data is being used.
Example Altmetric Score: https://www.altmetric.com/details/34049168
Provides a more modular framework to develop Flask Apps by delineating scope within different functions of the App. For example, the authorization portion can go in its own drop-in authorization app that is added to the main Flask App.
Sets up the ability to make our overall development more modular, and also provides a path to unit testing of the platform.
Basically, there is an init.py for every app we would like to add that defines a Flask Blueprint. For example:
In app/auth, the init.py will have:
from flask import Blueprint
auth_bp = Blueprint('auth',__name__,template_folder='../templates')
from app.auth import routes
and then in the app/init.py:
from app.auth import auth_bp
app.register_blueprint(auth_bp)
Modules like Flask-login are stubbed in the global space, and then a new function called create_app is used to create the full app.
I have an example of what this looks like for a role based authentication scheme using Flask-User at https://github.com/shots47s/NRC-Map-Converter-Flask
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.