cedadev/jasmin-manage

jasmin-manage

API for managing project resources on JASMIN.

Setting up a development environment

jasmin-manage requires at least Python 3.8, so you must first ensure a suitable Python version is installed.

First, check out the code:

git clone https://github.com/cedadev/jasmin-manage.git
cd jasmin-manage

Create and activate a new virtual environment and install:

python -m venv ./venv
source ./venv/bin/activate
pip install -e .

Install the local settings:

cp jasmin_manage_site/settings.py-local jasmin_manage_site/settings.py

Apply the migrations (this will create an SQLite database in the same directory as the code):

python manage.py migrate

Create a superuser to access the admin interface:

python manage.py createsuperuser

Then run the development server:

python manage.py runserver

The admin interface will then be available at http://localhost:8000/admin and the API will be available at http://localhost:8000/api (you must authenticate via the admin).

jasmin-manage's Issues

List GWSs by consortia or by funding source NCEO or NCAS

@mjpritchard, coming from a CEDA EO meeting today, there's an ask to be able to display a list of GWSs that are associated with NCEO. Is anything like this possible or at least to list GWSs belonging to a given consortia

Create filter on service API endpoint

We would like to be able to filter the services API endpoint by service name (not id) in order to get the details of the service and its requirements.

As part of this it probably also makes sense to expose the requirements under the self view of the service.

Add notification to staff when a requirement is submitted for provisioning

This could be email, but probably better notifying a slack channel

Email admins when a new requirement is moved to awaiting provisioning

This should send a notification to slack using a webhook.

Use per-requirement consortium mapping

Project has "default" consortium if known at creation time, but project creation without consortium is permitted.

At the point where a service is created, the project must either have a default consortium or one must be specified.

Add consortium column to Requirements admin list

useful to see (as well as filter by) consortia in this view
https://projects.jasmin.ac.uk/admin/jasmin_manage/requirement/?

add API endpoints for common queries

similar to what's been done with the accounts portal.

Have an any logged in user view of the projects with filters

Add per-project requirement number

In order to make referring to requirements within a project more natural, each requirement should have a "number in project" as well as an overall ID.

Add generic labelling to models

Allow arbitrary label/value pairs to be attached to model instances.

Add not before and not after dates for a resource

Allows compute rounds to be supported as it allows us to have an absolute amount as the total available.

Email notifications for project events

add capability to tag projects

would be nice to be able to tag projects as "ncas" "nceo" "ceh" "met office" etc (which may cut across consortia boundaries)

create flag "internal" on Consortium and "staff" on User

...so that we can exclude "internal" consortia from public view

• would still see & be able to create projects but via admin interface
• "staff" flag prevents non-staff from creating new projects in "internal" consortia.
• could still be added as collaborator on projects in "internal" consortia.

blank page instead of 403/404

Blank pages are shown when an incorrect page url is used, or when a page is accessed when the user isn't authenticated to use the page. This should throw 403/404 as required

Need ability to hide some categories of service

Example: "External Tenancy MCP", "Managed Tenancy MCP" are both deprecated, though we have current instances of them (which still need to be managed for now).
But when setting up a new project, we don't want users to be able to select them as an option.

Assumption in requirement creation that the units are in GB when creating a storage requirement

It would be good if this assumption is coded out and the frontend gets the unit from the back end, and enables unit selection based on the units of the resource.

Invite system for collaborators

The way the authentication works with OAuth 2.0 means that manage.jasmin doesn't know about a JASMIN user until they have signed in for the first time. This means only a user who has signed in can be added as a collaborator.

Also, having a list of all JASMIN users available to any authenticated user is potentially a privacy issue.

To get around this, we will use an "invite by email" system. The way this will work is:

• User creates a new project and becomes the owner
• Project owner invites a user to collaborate using their email address (which they will know if they are collaborating)
• The invite has a unique code that is sent to the email address as part of a link
• When the link is followed, the user following the link is forced to sign in (which creates the user record if it doesn't exist)
• The signed in user is associated with the project and role from the invite

Django REST Framework views for API

Multiple collaborators for project with roles

Roles will be Owner, Contributer and "Something readonly"

Hide the resources which have no quota when user is creating a requirement

Structured metadata for requirements

Allow metadata to be attached to requirements where the structure is defined by the resource.

Multiple managers for a consortium

blank page for "cedaproc" when viewing services

When going to the "cedaproc" services, the page goes blank instead of throwing an error or displaying the page.

Ability to class a new requirement as a replacement of an old one

This would allow a user to pick a requirement to replace the old one, so that changes on existing requirements can be handled better.

Currently new requirements can't be the total new requested size as this would mean that consortium managers cannot approve if the new total size puts them over quota, but the increase in size would not take them over quota.

Having the ability to mark something as a replacement would let us do logic for when a new volume is marked as PROVISIONED, to DECOMMISSION the old volume without having to remember what specifically needs doing in this case, and would allow a reduction in requirement to be handled. As well as handling the approval of new requirements properly

Edit requirements for a service

It would be useful for managers to be able to request changes by editing a specific requirement. For example, editing the amount of resource, or the end time for the requirement

Add reference field to project

Optional field for NERC award reference or similar.

Enable display and entry of quantities in sensible multiples

Lots of people enter e.g. 80 GB when they mean 80000 = 80 TB for a requirement. Can we make this easier (without changing the base unit)?

Start and end date for project

To be used as default start/end date for requirements to avoid entering every time.

Provide tooltips for icons

Icons in GUI need rollover tooltips to make their meaning more obvious.
Example: paper-plane icon representing "requested" status of a requirement.
See this conversation:
https://ncas-talk.slack.com/archives/C0UR29PPX/p1667903673615279

Expose paths of GWS by service.

Make REST API an OAuth 2.0 resource server

Protect the REST API with OAuth 2.0 scopes using django-oauth-toolkit.

This is required in order to implement the frontend as the frontend will be implemented as an OAuth 2.0 RP to this API.

Give staff users manager permissions for all consortia

Intro text on my projects defining process

Quick intro to how to create a project on the myprojects landing page and link to help doc. @mjpritchard to create some text

Does the user's email update when the user changes email in the accounts portal?

If this is complicated to solve, might be worth changing the user auth to match other jasmin services

Allow for comments when returning a project for changes

Make it clearer whether a requirement is for the delta or the new total

A requirement for 20 TB could be interpreted as 20 TB new space, or 20 TB total (on top of an existing provision). Need to agree a convention then stick to it and make it clear in the GUI. (we have a mixture of existing cases!)

Add resource chunks

Replace the total available for a resource with a sum over separate chunks of that resource.

Possibly allow requirements to be assigned to a chunk at provisioning time.

add start and end dates to consortium all_projects view

start (but especially end) dates would be really useful in the table of projects. Requested by an interested consortium manager.
If there's only space for one, please add end date. Ideally enable ordering too.

Need endpoint for listing services & their locations, by consortium

Need to be able to generate a text listing something like this:

select all requirement.location (e.g. volume path) for all services where service.category="Group Workspace" and where service.Project.consortium = X

While an admin view would be nice (as well) an API endpoint that can be consumed elsewhere is what's really needed, e.g. for the Storage Summary report. This needs all volumes by project & consortium, with their provisioned sizes (from here) which can then be matched programatically to volume paths & usage from the filesystem entries (now in Elastic Search, used for metrics).