usnistgov/corr

The Cloud of Reproducible Records

Overview

The Cloud of Reproducible Records (CoRR) is a web platform for storing and viewing metadata and data associated with simulation records for reproducibility and beyond. It is designed as a gateway for execution management tools that aim in capturing software executions. The need of those records for recalls, reproducibility, education, provenance and posterity is so critical now more than ever.

The platform is composed of five components. A database component which stores the meta-data about the diverse entities in relationship within the platform. An api component which identifies and allow software management tools to interact with the platform. A cloud component dedicated for the frontend component access to the meta-data and records. A storage component that handles non meta-data files management.

The CoRR platform is programming language agnostic, recording style and process independent. Instead of focusing in these specifics, the platform comes at a higher level by providing a unified gateway for execution management tools. They are the ones who handle these specifics. The platform focuses on networking records and scientists. Thus, making the long term survival of research reproducibility its core goal.

A CoRR instance is composed of five micro-services. The database micro-service which is a mongodb database combined with a python shared library corrdb for distributed entities models to the other micro-services. The api micro-service (corrapi) which is a restfull service that is the interface dedicated to execution management tools. The cloud micro-service which is a second restfull service (corrcloud) dedicated to the view micro-service (corrview) which is the frontend exposed to web users. Finally a nginx service takes care of unifying the access to the corrcloud, corrapi and corrview. Information on how to launch a CoRR instance is provided in LAUNCH.mdHow to use and also in videos (click on the corr youtube badge above.).

Citation

• Congo, F.Y.P., Wheeler, D., Hill, D.R.C. 2015. Building a Cloud Service for Reproducible Simulation Management. SciPy proceedings. 2015.
• Congo, F. Y. P. 2015. CoRR - Materials Genome Initiation (MGI) at NIST.

corr's Issues

Add gitter for support and development

Record and Project Privacy access

Add a toggle button to switch a project or record to be publicly or privately accessible.

Move corr-view media to S3

In the worry of keeping a clean repo we will have to move all the media to the S3 storage.

Ansible Management

• create ansible script to install well specified version numbers of dependencies
• create conda installation of corr -- no need for this
• remove install_requires from setup.py in all directories -- will be addressed in another issue #28 #29
• how to setup S3 -- will be addressed in another issue #28 #29
• test in new anaconda directory
• run in docker -- just to check the develop version -- all the apt-get's work
• update installation instructions
• remove config files in db, api, cloud and view that are no longer required

Setup ansible script to deploy on Docker instances

Have version number on front page

Show the code version number on the front page -- this will sometimes be a 0.1-dev_sha and other times be just a 0.1. Released versions of the code should not have the sha. This can all be automated using the get_version in the corrdb __init__.py. The version number is extracted from the git repository using the tags and SHAs.

Support file upload of ArchieML with no application

UI/UX CoRR Search

Design the result view of the Search action.
The result of the research should be grouped into: Users, Applications, Projects, Records, Diffs, Files, Comments. And other non private items. If the research is done while logged in the privacy scope stays at the logged user level. If done without being logged in, the research result should display only publicly accessible results.

UI/UX CoRR Navigation Bar

Figure out the Nav Bar content for v0.1

UI/UX CoRR Home

Freeze the home view design and link all the required actions to the backend by adding javascript functions, cloud service endpoints and material design components. Stormpath password requirements. Display errors in the view.

Use cases with Frontend

Find use cases for the frontend.
This is a room for the json and yaml records uploads from the frontend and also third parties integrations.

UI/UX CoRR Files

Provide a view for all the files of an objects to which files can be attached to.
It rather be a floating window like Comments.

Support file upload of YAML as a record with no application

Merge DDSM repos to keep the history

Frontend Dyanmic Upates

Automatic update of the frontend content with JS.

UI/UX CoRR Dashboard Diffs

Provide the list of diffs. In the list a short view of the Diffs is provided with some possible statistics. The same view can also display a detailed view of the diff by passing the diff id to the same url: /diffs/?id=diff-id or hide it in a REST POST request. Diffs can be downloaded.

Missing group field in corr-cloud

The group field to determine the account group: user, admin, unknown, developer is not taken care of. Which makes the registration of a user falls to unknown right now.

UI/UX CoRR Comments

For all the objects on which comments can be made on, provide a detailed display view for the comments. It can be a floating window or a separate web page.

UI/UX CoRR Dashboard Records

Provide the list of records. In the list a short view of the Records provided with some possible statistics. The same view can also display a detailed view of the record by passing the record label or id to the same url: /records/?label=record-label or /records/?id=record-id or hide it in a REST POST request. Records can be downloaded and diffed.

Use cases with API

Examples of simulations records using the API. This is where we have to think of integrating smartly some software management tools but also the corr-cmd tool.

Add production deployment with NGINX to Ansible script

• implement Jekyll / NGINX launch
• implement serve_db tag
• remove serve_develop tag
• don't store generated template files in corr repo

corr-cmd management tool

A software management tool that uses ansible and python utility libraries to record execution metadata and files. It is a great use case to show how to integrate a management tool to the CoRR backend. Another proper example can be provided with existing tools like sumatra, etc...

3rd party packages should not install in corr-db when simply running `python setup.py install`

Documentation

Setup the documentation for the code, the repository and the frontend.

Cloud and View Urls interdepency

Centralize the URLS in a single place in each sub-repo:
config.py for cloud and config.js for view.

Clean up running server processes in Ansible when developing

UI/UX CoRR Dashboard Base

Redesign the Dashboard so that we can have many views in it: Applications view, Projects views, Records views and Diffs views.

Fix corrdb.__version__ to have the from v0.1-dev_SHA

Access constraints

Enable User quota to 5Gb.
Enable maximum filesize for:

• user picture: 5 Mb.
• app logo: 5 Mb.

UI/UX CoRR Dashboard Applications

Provide the list of applications. In the list a short view of the Applications is provided with some possible statistics of how many projects and records was done with it. The same view can also display a detailed view of the application by passing the application name or id to the same url: /applications/?name=app-name or /applications/?id=app-id or hide it in a REST POST request.

All icons should have hover text to explain usage

Implement central test suite that runs all tests

• add pylint
• small automate test suite in db
• small automated test suite in cloud
• add coverage

Support file upload of JSON

Users should be able to upload a record on the front end without actually having a local application available.

Getting Started Page

Take users through the first steps in generating some records.

• implement some templating with moustache.js
• use showdown.js for translating markdown to html on the frontend
• take the user through the steps to generate, upload and download a simulation

UI/UX CoRR Dashboard Projects

Provide the list of projects. In the list a short view of the Projects is provided with some possible statistics. The same view can also display a detailed view of the project by passing the project name or id to the same url: /projects/?name=project-name or /projects/?id=project-id or hide it in a REST POST request. Projects can be downloaded.

Open up modal error display box with title, code and content.

The information returned to the user when the password is entered wrongly or is too short disappears to rapidly. A dialog box should open that the user clicks okay on so that the information can be read.

Set up instance so URL:PORT is no longer required

The port should not be required when giving the URL for the frontend view.

Likewise for the API

CoRR Web upload

Implement Project and Record upload. Also Project environment upload.
The button are added and greyed out.

Remove all hardwired URLs from frontend and move to config file

UI/UX CoRR Messages

Design a Messages management space for user messages: create, delete, view and be notified of new messages either by email or locally (number of unread messages).
This will require to change the current Message Model to add a read field to track if the message was read or not: capital to generated (number of unread messages).

clean setup

Clean up the corr central repo for well controlled clean setup of the platform.
This issue has to be solved to better serve the upcoming ansible management setup.

Switch to Python3

No-App capability for the API

Allow projects and records push from API without an application.

Create an automated test suite for the API.

This does not need to be complete. Just a few use cases for the current milestone.

Fix text on front page

Email from Stormpath for password reset should be from the "CoRR team" and not "Yannick Congo"

create conda package for corr

Make Ansible deploy on AWS servers

Use CI tool with test suite

API v0.1 review

Review the api endpoints for corr version 0.1 release.